- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates why vision-language-action (VLA) models fail when target objects are displaced from their training-time positions. Using activation patching, the authors show that information about the new object location is present in visual representations but is not effectively utilized by the policy. They propose a training-free method that identifies attention heads responsible for transmitting spatial information to action tokens and amplifies their input-dependent activations. Experiments on five VLA architectures demonstrate improved closed-loop success in three models, with additional validation on LIBERO-Pro. The study combines mechanistic analysis with a practical inference-time intervention.
Critical strengths
-
Meaningful mechanistic contribution. The paper goes beyond documenting VLA failures under spatial perturbations by investigating their internal causes. The activation-patching experiments distinguish the presence of spatial information from its effective use and identify specific attention heads involved in transmitting object-displacement information to action predictions. This provides useful insights into how spatial information is processed by VLA architectures.
-
Novel training-free intervention with controlled validation. The proposed attention-head amplification improves closed-loop success without modifying model parameters. The improvements are substantial for some models, particularly π0 and π0.5, and are supported by random-head, deviation-matched, and energy-matched controls. The use of separate validation and test states, together with transfer experiments on LIBERO-Pro, strengthens the empirical evidence.
Critical weaknesses
-
Limited generality of the proposed intervention. Although spatial information is identified in all five architectures, amplification improves performance in only three, with statistically significant gains on held-out test states clearly demonstrated for pi0 and pi0.5. Furthermore, the method is evaluated primarily on a narrow perturbation setting where a single target object is displaced from a fixed training-time position. It remains unclear whether the identified attention heads and amplification strategy generalize to more complex spatial changes, multiple moving objects, or unseen manipulation tasks.
-
Practical applicability and sensitivity to hyperparameters. Despite being training-free, the method requires extensive activation-patching experiments, clean-scene reference activations, and closed-loop validation to select the number of heads and amplification factor. The authors also report a narrow range of effective amplification factors, with stronger interventions sometimes degrading performance. These requirements limit the practicality of applying the method to previously unseen tasks or physical robots without access to controlled calibration environments.
Questions for authors
-
Generalization across tasks. The experiments primarily consider moving one target object while keeping the remaining scene unchanged. Would the same attention heads remain relevant when multiple objects change positions or when the target object appears in previously unseen scene configurations? This would help establish whether the identified heads encode general spatial relationships or task-specific displacement information.
-
Statistical robustness. Table 1 reports significant improvements for OpenVLA-OFT on the combined evaluation set, but the held-out test gain is smaller and not marked as statistically significant. Can the authors provide additional independent evaluations or confidence intervals to clarify the reliability of improvements across all three successful architectures?
-
Deployment requirements. How much task-specific calibration is required to identify the attention heads and select amplification factors? Can heads identified on one manipulation task be reused on other tasks without additional activation patching or closed-loop validation? This question is particularly important for assessing the practical value of the training-free claim.
3 comments
Thank you again for your review. As the discussion period closes on November 18, we would be grateful to know whether our response and the revised PDF address your concerns. Your question about multiple objects and unseen scenes led to the non-target experiments (App. T) and the transfer, without re-screening, to LIBERO-Pro and to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 (App. V); your question about deployment requirements is answered with the screening cost (App. K), the magnitude rule (App. Q) and the 20-episode clean-scene check (App. V). If any point remains open, we will respond before the deadline.
Some of my quesions are still opened, but I'm ready to increase my score.
We thank the reviewer for a careful review and for questions that we could turn into experiments. We answer each point and give the location of the change (blue text in the revised PDF).
Multiple objects and unseen scenes (W1, Q1). This question led to a new experiment (App. T). We moved non-target objects on a grid, alone and together with the target. The screened heads carry the displacement of a non-target object as well, and the amplification is not target-selective: when only a non-target moves, OpenVLA-OFT loses 6.4 points at 10 cm (random heads +0.6), π0 and π0.5 change by −3.2 and −0.6 (not significant); when both objects move, π0 gains 9.0 points over 40 states (p=.008) and π0.5 12.3, while OpenVLA-OFT loses 2.5. Together with App. S, where the top head of π0 also responds when a camera rotation shifts the object in the image, this says that the heads encode a change of object position in the image rather than a task-specific displacement. §4.4 ("Scope of the lever") and the Limitations now state the non-selectivity.
On unseen scenes and tasks: the heads were screened on LIBERO-Object only and applied without re-screening to LIBERO-Pro (Table 2) and, new in App. V, to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 with the same checkpoints. π0.5 gains 10.6 points on LIBERO-Goal (p=.005) and is neutral on LIBERO-10; π0 is neutral on all three. On a π0.5 checkpoint fine-tuned on ManiSkill3 (App. W) the screen finds the same four heads, so the routing heads are a property of the architecture and survive fine-tuning, but the amplification does not transfer. Heads therefore reuse across tasks; the lever does not reuse across checkpoints.
Generality and the perturbation setting (W1). We agree that the mechanism is established for one axis, and we list this first in the Limitations. We chose a single controlled variable deliberately: paired clean and moved scenes are what allow the patching, the screen and the intervention to act on the same quantity and support a causal statement. We also agree that the broader question matters, so App. S tests whether the same heads carry camera translation and rotation responses (they carry at most 3% in OpenVLA-OFT) and App. U applies the deployed settings to the other LIBERO-Plus families (neutral for OpenVLA-OFT and π0, −2.6 for π0.5).
Statistical robustness (Q2). We added a cluster bootstrap interval and a sign-flip permutation test over initial states for every arm (App. O) and 19 further held-out states for π0 and OpenVLA-OFT. Held-out: π0 +8.0 [+2.7,+13.8], p=.010; π0.5 +14.7 [+2.9,+26.0], p=.039; OpenVLA-OFT +3.8 [−0.4,+8.2], p=.13, positive in each of three held-out estimates but not significant, which the abstract, §4.4 and §5 now state; "three of five" is removed. App. P bounds run-to-run variability (800 of 800 outcomes reproduced with seeded noise; 91 to 97% agreement without, with symmetric flips).
Practical applicability and deployment requirements (W2, Q3). We agree, and after reading this review and the others we changed what the paper claims to be. The contribution is the analysis: the displaced position is present (decodable to within a few centimetres, App. L), under-used (the first grasp follows 20 to 65% of it, App. N) and routed by a few heads; the amplification is the causal test of that account. The title, abstract, §1 and §5 say so, and we quantify the lever openly (4 to 25% of the gap, App. R). On the calibration required: the screen costs 1.5 to 2.8 GPU-hours per policy (App. K), the magnitude follows a rule fixed before any intervention (§3.4, App. Q), and App. V gives a 20-episode clean-scene check (first-closure shift at most 1 cm) that is a necessary no-harm condition across 15 settings, with a post hoc threshold, which we state. We do not claim the method is ready for a physical robot; §5 names that as the next step.
We are grateful for a review that pushed us to measure the scope of the result instead of asserting it; the paper is better defined for it.