- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a new method for ICIL for VLA models as they suffer from OOD called StellaVLA. StellaVLA has two expert heads during training, one which predicts actions and one which predicts supervised language output which is task related (subtask description, 2D trace, 3D motion). This language expert is dropped during inference and the supervision data for it is produced using an LLM on the given training data. StellaVLA achieves high results on LIBERO, LIBERO-Plus, VLA-Arena and real robot tasks. Ablations show that the model needs either no or correct in-context conditioning or the performance is dropping heavily.
Critical strengths
- Experiments include a wrong-task intervention test which shows that the method is actually relying on the given demonstrations
- The overall structure of the paper is easy to follow
- Very good results on the trained models on different benchmarks and real robot tasks
Critical weaknesses
Weaknesses
- Experiments are not part of the contribution as they are necessary to show that the proposed method is working.
- The claim that OOD robustness comes from the dual training setup is not given, as shown in Table 8 in the appendix and should be reframed in the paper.
- The paper says that ICIL methods usually only look at what an expert did, but now why. It is not clear how StellaVLA does this differently and if it is evaluated.
- The StarVLA-OFT baseline removes the retrieved demonstration and the spatial-language supervision. This makes it unclear which of these two actually contributes to the proposed method, as neither baseline is evaluated.
- It is not clear how retrieval works for LIBERO-Plus and the VLA-Arena L1/L2. Especially in regards to the language descriptions. Which demonstrations were used and how do the instructions differ from the demos used for training?
- It is not clear how StellaVLA differs from the LAP (Zha et al.) model. Differentiating it would strengthen the contribution and novelty of the paper. Also including a related work section about embodied reasoning as auxiliary supervision without decoding it at inference would be helpful to put this work into perspective.
Minor Weaknesses
- Line 81 the word Context is written with a capital letter in the beginning but nothing else of “in-context structured distillation”
- It is not clear if the evaluated models are only one seed or multiple ones
- It is not clear how well the Qwen model can divide the tasks into subtasks and it is not evaluated against a hand-labeled control sub-dataset
- There is also more related work which should be cited and included in the paper to compare to the proposed method, like Knowledge Insulating VLAs (Driess et al.) and ECoT-Lite (Chen et al.) and other work in this direction.
Questions for authors
- Is the semantic segmentation evaluated somehow using a hand annotated validation set?
- Why is the success for some VLA-Arena L0 tasks lower compared to the L1 and L2 variants?
- How do the demos differ from VLA-Arena L0 and L1/L2?
- What is the difference from StellaVLA to LAP?
Please also consider all weaknesses also as a question that should be answered.