- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the robustness of Vision-Language-Action (VLA) models under spatial object perturbations, where policies may continue reaching toward an object's original position after it has been displaced. The authors use activation patching to examine how object-position information propagates through visual tokens and attention heads, and propose a training-free attention-head amplification method to strengthen the contribution of displacement-sensitive heads to action prediction. The method identifies relevant attention heads through counterfactual interventions and amplifies their input-dependent deviations without modifying model parameters.
Critical strengths
- The paper investigates the internal mechanisms underlying spatial robustness failures in VLA models. Through activation patching and attention-head interventions, the authors examine whether object-position information is absent from model representations or insufficiently utilized during action prediction. This analysis provides empirical evidence about how spatial information influences VLA behavior.
- The paper evaluates the proposed intervention across multiple VLA architectures. The experiments cover five VLA policies with different action-generation mechanisms and include paired closed-loop evaluations, attention-head controls, and transfer experiments on LIBERO-Pro. These comparisons help characterize the effectiveness and limitations of the proposed method.
Critical weaknesses
- The proposed method primarily adapts established activation patching and attention-head intervention techniques to VLA models. Although the application to spatial perturbations provides a different setting, the core amplification mechanism is relatively straightforward. The paper should better clarify its methodological novelty compared with existing activation steering and inference-time attention manipulation methods.
- The experiments focus on target-object displacement in simulated robotic manipulation tasks. It remains unclear whether the identified attention-head mechanisms generalize to other perturbations, such as viewpoint changes, distractors, object appearance variations, or more complex manipulation scenarios. Additional evaluations would strengthen the generalizability claims.
- Attention-head amplification improves closed-loop success in only three of the five evaluated policies, while providing no consistent gains for OpenVLA and GR00T N1.7. These results suggest that the method depends heavily on architecture-specific information routing. A more systematic characterization of the conditions under which amplification succeeds or fails would strengthen its practical applicability.
- The proposed intervention requires selecting specific attention heads, amplification factors, and, in some cases, displacement subspaces. The paper acknowledges that effective amplification ranges can be narrow and vary across architectures. However, it remains unclear how reliably these configurations can be identified for unseen tasks, checkpoints, or deployment environments without extensive validation.
- Although the method improves success rates in several settings, the gains remain relatively modest in some cases. For example, OpenVLA-OFT improves from 12.3% to 17.9% at a 10 cm displacement, leaving most perturbed episodes unsuccessful. Moreover, not all held-out improvements are statistically significant. Further experiments are needed to demonstrate whether the improvements are sufficiently reliable for practical robotic deployment.
Questions for authors
Please see the above comments.
[1/2] Official Rebuttal
Thank you for your constructive review and for highlighting our two central findings: (1) that blackout and freezing produce distinct physical behaviors, and (2) that higher task success does not by itself reduce physical side effects. Below, we present 80 new post-closure WidowX trials and pre-clipping command analyses showing that drops, non-target contact, and freezing-induced joint-limit proximity all appear on the physical robot, while clarifying the scope of the simulation JVE and drop claims in the revised manuscript.
[W1] Drops after gripper closure on the robot.
We ran 80 additional WidowX trials with the fault starting after gripper closure (π0.5, ten trials per cell; Table R5).
Table R5. WidowX task success and drops in ten post-closure trials per condition.
Under wrist faults, drops occur in 5 of 20 trials without proprioception and in 1 of 20 with it (0 of 20 under clean inputs), matching the simulation finding that post-closure drops under wrist faults are concentrated in policies without proprioception. On the same WidowX block-in-bowl task, we also observed non-target contacts (Section 2.4) and found that the arm approached its joint limits under post-closure scene freezing, as detailed below. The full results are reported in Appendix A.3 of the revision.