- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether self-attention is needed in VLA action expert by ablating the attention layers during training and inference time, and find that self-attention may not be necessary on some tasks.
Critical strengths
The paper studies the necessity of self-attention in action experts
Critical weaknesses
- The core research question whether post-hoc ablation loss implies a module must be retained during training has been well studied in model pruning, which the paper also cites. Re-confirming the conclusion for SA in two VLA action experts is useful but delivers very limited new information and novelty.
- The success rates on a few LIBERO tasks where some saturated at 100% show these tasks are solvable even without SA. It does not show SA is unnecessary for broader tasks, larger datasets, or settings that need a more expressive expert. Stronger benchmarks, and a well-tuned baseline are needed before the general conclusion is drawn. Additionally, the dense SmolVLA baseline (76.4% mean) seems to be below publicly reported results around 80%-90%. Without a faithful reproduction, the conclusion can be inconvincing.
Questions for authors
See weaknesses above.
2 comments
Reviewer Responses
We thank the authors for the new experiments.
- The reviewer's concern regarding the contribution and novelty has not been resolved. The authors suggest this work is a "VLA-specific contribution that combines reproducible performance gains with evidence about retained computation", which seems to be only empirical, while the methodology is stemmed from existing model pruncing studies.
- Furthermore, the applicability to SmolVLA does not suggest conclusions to broad VLA models. The reviewer also expects more insights regarding why no-SA imrpoves performance particularly on LIBERO-Plus, and if the improvements hold for other VLA models in general.
I keep the rating for now.
Response to Reviewer CRQW
Thank you. We revised the paper around a concrete architectural finding: matched downstream adaptation produces effective action experts with designated SA branches bypassed and cross-attention retained.
1. Contribution beyond the pruning principle. Pruning provides the methodological foundation (Section 7). Our VLA-specific contribution combines reproducible performance gains with evidence about retained computation. At ten integration steps and ten executed actions (S10/H10), three independent SmolVLA training seeds yield a 13.58-point No-SA gain, while deleting SA from the corresponding Dense checkpoints yields 1.15% success (Table 1). Freezing the entire VLM preserves a 6.62 ± 0.38-point advantage with the expert and external projections trainable (Table 4). Fixed-policy two-step interventions further show that adding early SA reverses its observed contribution when late SA is retained (Section 5.1). These findings characterize both effective simplified experts and conditional SA use in closed-loop control.
2. Stronger Dense reference. We reevaluated
lerobot/smolvla_liberoalongside our matched models. All use S10/H10 within each evaluation protocol:Values are success percentages; ± denotes sample SD across three independent training seeds. Public Dense is one published checkpoint. Our matched pair uses batch 16 and a 30k decay horizon; the public configuration records batch 32 and 25k decay, with different input/preprocessing settings. Both use 25k updates. Matched retraining under the public recipe remains untested. Tables 2, 8, and 10 distinguish the controlled architectural comparison from this external reference.
3. Saturation and perturbations. On clean LIBERO-Long, matched No-SA improves success from 56.67% to 83.82% (Appendix A, Table 8). The Plus evaluation covers all 10,030 variants across four suites and seven perturbation categories, extending the matched-recipe gain to these perturbations.
GR00T's 5,037 Long/Object variants give 73.95/63.15/72.23% for Dense/deletion/No-SA, respectively: adaptation recovers 9.07 points over deletion (Table 3). Each suite has one adapted checkpoint pair. We corrected the clean-GR00T deletion attribution: 85.33% came from the public checkpoint (Appendix B). The revised comparison uses matched Dense/deletion weights.
The claim concerns downstream adaptation in these two interleaved architectures. Plus tests broader perturbations within LIBERO's simulator and base tasks. The revised introduction organizes the contribution around effective simplified experts, conditional computation, and measured deployment value.