- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces PermVLA, a method import permutation language modelling into VLA. Unlike autoregressive prediction, PermVLA creates subsets from an action chunk with permutation for training. The experiments are carried out on LIBERO dataset. Additional evaluation for zero shot is performed on LIBERO-Plus and cross task evaluation on CALVIN.
Critical strengths
- Compared to autoregressive modelling, PermVLA shows improvement on LIBERO, LIBERO-Plus and CALVIN.
Critical weaknesses
-
The permutation ignores the temporal dependencies between actions, this potentially hurts performance on long-horizon tasks.
-
The experimental results are not convincing. Multiple baselines (pi0, SmolVLA, WorldVLA and RIPT-VLA) outperforms PermVLA on LIBERO. The strongest baseline i.e. RIPT-VLA outperforms PermVLa around 10% on spatial, object and goal tasks in LIBERO, and is around 60% on long horizon task. Same on the zero shot setting, RIPT-VLA significantly outperforms PermVLA. On the cross task setting, OpenVLA significantly outperforms PermVLA on all metrics.
Questions for authors
What is the motivation to use permutation modelling given the actions have temporal dependencies and they are critical?