acceptodds
Under review as a conference paper at ICLR 2027

Improving the Robustness of VLAs to Spatial Object Perturbation via Attention-Head Amplification

Abstract

Vision-Language-Action (VLA) policies struggle under small perturbations, which limits robotic deployment. We focus on spatial object perturbation. When the target object is displaced from its training-time position, the policy fails to manipulate it and often reaches toward the training-time position. With activation patching we show that this failure reflects non-use of the object's position rather than its absence, and we propose a training-free method that mitigates it by amplifying the few attention heads that carry that position to the action tokens. On OpenVLA, OpenVLA-OFT, , , and GR00T N1.7 on LIBERO-Object, the new object location is present in a few object patches of the input visual tokens and is consumed in a limited band of layers. We then test every attention head one at a time. We run the policy on the scene with the moved object, but feed that head the output it would produce if the object were still at its training-time position, and measure how much the planned action stops following the object. We found that in every policy a small number of heads carry the displacement. Amplifying the input-dependent deviation of these heads raises paired closed-loop success, at 10 cm from 12.3% to 17.9% in OpenVLA-OFT and from 29.1% to 41.8% in , and at 15 cm from 42.1% to 50.5% in . The gain reproduces on the test states and transfers to LIBERO-Pro.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action-models, Out-of-distribution robustness, Mechanistic interpretability, Activation patching, Activation steering

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_prrxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why vision-language-action (VLA) models fail when target objects are displaced from their training-time positions. Using activation patching, the authors show that information about the new object location is present in visual representations but is not effectively utilized by the policy. They propose a training-free method that identifies attention heads responsible for transmitting spatial information to action tokens and amplifies their input-dependent activations. Experiments on five VLA architectures demonstrate improved closed-loop success in three models, with additional validation on LIBERO-Pro. The study combines mechanistic analysis with a practical inference-time intervention.

Critical strengths
  1. Meaningful mechanistic contribution. The paper goes beyond documenting VLA failures under spatial perturbations by investigating their internal causes. The activation-patching experiments distinguish the presence of spatial information from its effective use and identify specific attention heads involved in transmitting object-displacement information to action predictions. This provides useful insights into how spatial information is processed by VLA architectures.

  2. Novel training-free intervention with controlled validation. The proposed attention-head amplification improves closed-loop success without modifying model parameters. The improvements are substantial for some models, particularly π0 and π0.5, and are supported by random-head, deviation-matched, and energy-matched controls. The use of separate validation and test states, together with transfer experiments on LIBERO-Pro, strengthens the empirical evidence.

Critical weaknesses
  1. Limited generality of the proposed intervention. Although spatial information is identified in all five architectures, amplification improves performance in only three, with statistically significant gains on held-out test states clearly demonstrated for pi0 and pi0.5. Furthermore, the method is evaluated primarily on a narrow perturbation setting where a single target object is displaced from a fixed training-time position. It remains unclear whether the identified attention heads and amplification strategy generalize to more complex spatial changes, multiple moving objects, or unseen manipulation tasks.

  2. Practical applicability and sensitivity to hyperparameters. Despite being training-free, the method requires extensive activation-patching experiments, clean-scene reference activations, and closed-loop validation to select the number of heads and amplification factor. The authors also report a narrow range of effective amplification factors, with stronger interventions sometimes degrading performance. These requirements limit the practicality of applying the method to previously unseen tasks or physical robots without access to controlled calibration environments.

Questions for authors
  1. Generalization across tasks. The experiments primarily consider moving one target object while keeping the remaining scene unchanged. Would the same attention heads remain relevant when multiple objects change positions or when the target object appears in previously unseen scene configurations? This would help establish whether the identified heads encode general spatial relationships or task-specific displacement information.

  2. Statistical robustness. Table 1 reports significant improvements for OpenVLA-OFT on the combined evaluation set, but the held-out test gain is smaller and not marked as statistically significant. Can the authors provide additional independent evaluations or confidence intervals to clarify the reliability of improvements across all three successful architectures?

  3. Deployment requirements. How much task-specific calibration is required to identify the attention heads and select amplification factors? Can heads identified on one manipulation task be reused on other tasks without additional activation patching or closed-loop validation? This question is particularly important for assessing the practical value of the training-free claim.

3 comments

We thank the reviewer for a careful review and for questions that we could turn into experiments. We answer each point and give the location of the change (blue text in the revised PDF).

Multiple objects and unseen scenes (W1, Q1). This question led to a new experiment (App. T). We moved non-target objects on a grid, alone and together with the target. The screened heads carry the displacement of a non-target object as well, and the amplification is not target-selective: when only a non-target moves, OpenVLA-OFT loses 6.4 points at 10 cm (random heads +0.6+0.6), π0\pi_0 and π0.5\pi_{0.5} change by −3.2-3.2 and −0.6-0.6 (not significant); when both objects move, π0\pi_0 gains 9.0 points over 40 states (p=.008p=.008) and π0.5\pi_{0.5} 12.3, while OpenVLA-OFT loses 2.5. Together with App. S, where the top head of π0\pi_0 also responds when a camera rotation shifts the object in the image, this says that the heads encode a change of object position in the image rather than a task-specific displacement. §4.4 ("Scope of the lever") and the Limitations now state the non-selectivity.

On unseen scenes and tasks: the heads were screened on LIBERO-Object only and applied without re-screening to LIBERO-Pro (Table 2) and, new in App. V, to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 with the same checkpoints. π0.5\pi_{0.5} gains 10.6 points on LIBERO-Goal (p=.005p=.005) and is neutral on LIBERO-10; π0\pi_0 is neutral on all three. On a π0.5\pi_{0.5} checkpoint fine-tuned on ManiSkill3 (App. W) the screen finds the same four heads, so the routing heads are a property of the architecture and survive fine-tuning, but the amplification does not transfer. Heads therefore reuse across tasks; the lever does not reuse across checkpoints.

Generality and the perturbation setting (W1). We agree that the mechanism is established for one axis, and we list this first in the Limitations. We chose a single controlled variable deliberately: paired clean and moved scenes are what allow the patching, the screen and the intervention to act on the same quantity and support a causal statement. We also agree that the broader question matters, so App. S tests whether the same heads carry camera translation and rotation responses (they carry at most 3% in OpenVLA-OFT) and App. U applies the deployed settings to the other LIBERO-Plus families (neutral for OpenVLA-OFT and π0\pi_0, −2.6-2.6 for π0.5\pi_{0.5}).

Statistical robustness (Q2). We added a cluster bootstrap interval and a sign-flip permutation test over initial states for every arm (App. O) and 19 further held-out states for π0\pi_0 and OpenVLA-OFT. Held-out: π0\pi_0 +8.0+8.0 [+2.7,+13.8][+2.7,+13.8], p=.010p=.010; π0.5\pi_{0.5} +14.7+14.7 [+2.9,+26.0][+2.9,+26.0], p=.039p=.039; OpenVLA-OFT +3.8+3.8 [−0.4,+8.2][-0.4,+8.2], p=.13p=.13, positive in each of three held-out estimates but not significant, which the abstract, §4.4 and §5 now state; "three of five" is removed. App. P bounds run-to-run variability (800 of 800 outcomes reproduced with seeded noise; 91 to 97% agreement without, with symmetric flips).

Practical applicability and deployment requirements (W2, Q3). We agree, and after reading this review and the others we changed what the paper claims to be. The contribution is the analysis: the displaced position is present (decodable to within a few centimetres, App. L), under-used (the first grasp follows 20 to 65% of it, App. N) and routed by a few heads; the amplification is the causal test of that account. The title, abstract, §1 and §5 say so, and we quantify the lever openly (4 to 25% of the gap, App. R). On the calibration required: the screen costs 1.5 to 2.8 GPU-hours per policy (App. K), the magnitude follows a rule fixed before any intervention (§3.4, App. Q), and App. V gives a 20-episode clean-scene check (first-closure shift at most 1 cm) that is a necessary no-harm condition across 15 settings, with a post hoc threshold, which we state. We do not claim the method is ready for a physical robot; §5 names that as the next step.

We are grateful for a review that pushed us to measure the scope of the result instead of asserting it; the paper is better defined for it.

Thank you again for your review. As the discussion period closes on November 18, we would be grateful to know whether our response and the revised PDF address your concerns. Your question about multiple objects and unseen scenes led to the non-target experiments (App. T) and the transfer, without re-screening, to LIBERO-Pro and to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 (App. V); your question about deployment requirements is answered with the screening cost (App. K), the magnitude rule (App. Q) and the 20-episode clean-scene check (App. V). If any point remains open, we will respond before the deadline.

Some of my quesions are still opened, but I'm ready to increase my score.

Reviewer_iEyiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness of VLA models under spatial object perturbations, where policies may continue reaching toward an object's original position after it has been displaced. The authors use activation patching to examine how object-position information propagates through visual tokens and attention heads, and propose a training-free attention-head amplification method to strengthen the contribution of displacement-sensitive heads to action prediction. The method identifies relevant attention heads through counterfactual interventions and amplifies their input-dependent deviations without modifying model parameters.

Critical strengths
  1. The paper investigates the internal mechanisms underlying spatial robustness failures in VLA models. Through activation patching and attention-head interventions, the authors examine whether object-position information is absent from model representations or insufficiently utilized during action prediction. This analysis provides empirical evidence about how spatial information influences VLA behavior.
  2. The paper evaluates the proposed intervention across multiple VLA architectures. The experiments cover five VLA policies with different action-generation mechanisms and include paired closed-loop evaluations, attention-head controls, and transfer experiments on LIBERO-Pro. These comparisons help characterize the effectiveness and limitations of the proposed method.
Critical weaknesses
  1. The proposed method primarily adapts established activation patching and attention-head intervention techniques to VLA models. Although the application to spatial perturbations provides a different setting, the core amplification mechanism is relatively straightforward. The paper should better clarify its methodological novelty compared with existing activation steering and inference-time attention manipulation methods.
  2. The experiments focus on target-object displacement in simulated robotic manipulation tasks. It remains unclear whether the identified attention-head mechanisms generalize to other perturbations, such as viewpoint changes, distractors, object appearance variations, or more complex manipulation scenarios. Additional evaluations would strengthen the generalizability claims.
  3. Attention-head amplification improves closed-loop success in only three of the five evaluated policies, while providing no consistent gains for OpenVLA and GR00T N1.7. These results suggest that the method depends heavily on architecture-specific information routing. A more systematic characterization of the conditions under which amplification succeeds or fails would strengthen its practical applicability.
  4. The proposed intervention requires selecting specific attention heads, amplification factors, and, in some cases, displacement subspaces. The paper acknowledges that effective amplification ranges can be narrow and vary across architectures. However, it remains unclear how reliably these configurations can be identified for unseen tasks, checkpoints, or deployment environments without extensive validation.
  5. Although the method improves success rates in several settings, the gains remain relatively modest in some cases. For example, OpenVLA-OFT improves from 12.3% to 17.9% at a 10 cm displacement, leaving most perturbed episodes unsuccessful. Moreover, not all held-out improvements are statistically significant. Further experiments are needed to demonstrate whether the improvements are sufficiently reliable for practical robotic deployment.
Questions for authors

Please see the above comments.

2 comments

We thank the reviewer for a thorough review. We answer the five weaknesses in turn and give the location of each change (blue text in the revised PDF).

Methodological novelty (W1). We agree that the form of the edit, a scalar gain on a head output, is simple, and we now say what is and is not new in §3.4. Activation steering adds a direction estimated from contrastive prompt pairs (Turner et al., 2023) or from a probe trained on labeled examples (Li et al., 2023); attention recalibration reweights attention maps (Zhang et al., 2026). Our intervention fits no direction and changes no attention weight: it scales the input-dependent deviation of the head output that the causal screen itself identified, so the quantity screened and the quantity edited are the same, and the closed-loop result tests the screen's account. After reading this review we also changed the framing of the paper. The contribution is the diagnosis (position present, under-used, routed by a few heads), and the amplification is its causal test; the title, abstract, §1 and §5 now say so. App. M adds the mechanism the edit acts on: in OpenVLA-OFT two screened heads read the object's wrist-camera patches with attention weight .34 and .33 in the clean scene and follow the moved object with only .15 and .19, so what the policy lacks is not the position but the weight with which it is read.

Other perturbations (W2). We agree that the mechanism is established for one axis and we list this first in the Limitations. We chose a single controlled variable deliberately, because paired clean and moved scenes are what allow the patching, the screen and the intervention to act on the same quantity and support a causal statement; a study that varied several axes at once could not have made that statement. We also agree that the direction the reviewer points to is the right one, so we ran what could be run. App. S tests axis specificity: in OpenVLA-OFT the screened heads carry at most 3% of the response to a camera translation or rotation, carried instead by a disjoint set of heads, and in π0\pi_0 the top head responds when a rotation shifts the object in the image, so the heads respond to the object's image position. App. T moves non-target objects (the amplification is not target-selective; OpenVLA-OFT loses 6.4 points when only a distractor moves). App. U applies the deployed settings to the other LIBERO-Plus families (neutral for OpenVLA-OFT and π0\pi_0, −2.6-2.6 points for π0.5\pi_{0.5}). §4.4 summarises these in a new paragraph "Scope of the lever".

Three of five policies and the conditions for success (W3). We now characterise the two failures at the head level (App. M). Decodability does not separate the policies (App. L). In OpenVLA the routing is distributed: the write of the screened heads is a minor part of the residual change (cosine .08 to .32, against .72 in OpenVLA-OFT), so scaling four heads scales a minority of the signal. In GR00T the two heads of the last DiT block write three to four times more toward −x-x than +x+x, and amplification enlarges that imbalance. App. V gives a measurable condition: a first-closure shift of at most 1 cm on 20 clean episodes is necessary for the absence of harm (0 of 8 settings harmed below it, 4 of 7 above, across six checkpoints and three further LIBERO suites); it is not a predictor of gain, and its threshold is post hoc, which we state. With cluster-aware statistics on held-out states the count is now "two significant, one consistent but not significant" (§4.4, §5).

Selecting heads, factors and subspaces (W4). §3.4 now states the rule that fixed the reported magnitude, App. Q gives the full sweep, and App. B the candidate sets. We tested transfer of the configuration to a fine-tuned checkpoint (App. W): the screen finds the same four heads of π0.5\pi_{0.5} on a ManiSkill3 checkpoint, but the amplification does not transfer, so a configuration belongs to a checkpoint. We state this in §5, and the 20-episode clean-scene check of App. V is the cheapest test we can offer before deployment.

Modest gains and significance (W5). We agree. App. R gives the fraction of the clean-to-displaced gap closed, 4 to 25%, against 45 to 59% for a LoRA on the backbone with the encoder frozen, and the held-out results with cluster intervals are π0\pi_0 +8.0+8.0 [+2.7,+13.8][+2.7,+13.8] (p=.010p=.010), π0.5\pi_{0.5} +14.7+14.7 [+2.9,+26.0][+2.9,+26.0] (p=.039p=.039), OpenVLA-OFT +3.8+3.8 [−0.4,+8.2][-0.4,+8.2] (p=.13p=.13). We no longer present the amplification as sufficient for deployment; we present it as evidence for the diagnosis.

Ethics flag. The review flags the paper for ethics review. All experiments run public checkpoints on public simulation benchmarks and involve no human subjects or data; we would be glad to address any specific concern the reviewer has in mind.

We thank the reviewer again; the review led directly to the clearer statement of novelty and scope in the revision.

Thank you again for your review. As the discussion period closes on November 18, we would be grateful to know whether our response and the revised PDF address your concerns. Your question about what distinguishes the intervention from activation steering and attention manipulation led to a precise statement in §3.4, and your question about the selection of heads, factors and subspaces led us to state the magnitude rule (§3.4, App. Q) and to report every candidate set (App. B). Other perturbation families are tested in App. S and U, and the paper now presents the amplification as the causal test of the diagnosis rather than as a remedy. If any point remains open, we will respond before the deadline.

Reviewer_mkZ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies spatial object perturbations in frozen VLA policies. It uses matched clean/moved-scene activation patching to identify attention heads that affect displacement responses, then amplifies their deviations from clean-scene means. The study covers five policies, finds closed-loop improvements in three, and evaluates transfer to LIBERO-Pro, entirely in simulation.

Critical strengths
  1. A focused mechanistic question with interventions that go beyond attention visualization. The same head outputs are used for screening and amplification, connecting the analysis to a practical intervention without updating policy weights.
  2. Careful evaluation includes paired outcomes, held-out test states, random and magnitude-matched controls, transfer experiments, and substantial failure analysis. Table 1 reports significant test-set gains of 10.4 and 14.7 percentage points for π0 and π0.5, respectively.
Critical weaknesses
  1. The demonstrated practical benefit is conditional. Amplification does not reliably improve OpenVLA or GR00T, while Appendix H shows a narrow useful amplification range: larger factors can reduce success and harm smaller displacements. These limitations restrict its value as a robustness remedy, even though the mechanistic findings are informative.
  2. All evaluations are limited to LIBERO-Object and LIBERO-Pro simulations. Section 5 acknowledges that usable heads have not been verified after fine-tuning. Effectiveness on physical robots and hardware-adapted policies therefore remains unverified, leaving a substantial gap between the simulation results and practical robotic use.
  3. Held-out evidence varies. OpenVLA-OFT improves by 5.6 points when validation and test states are pooled, but only 3.7 on test states, without a significance marker. Multiple displacement conditions share an initial state, but cluster-aware uncertainty for closed-loop success is not reported. Appendix A's state-level bootstrap concerns patching measurements rather than these success gains.
  4. Some causal interpretations exceed the measurements. T(0)=1 follows from replacing all visual tokens with clean-scene values and does not alone establish fully preserved, accurate position information. A narrower interpretation or absolute displacement-tracking measurements would better support the routing explanation.
Questions for authors
  1. Can you validate the screening and amplification on a physical robot using a hardware-adapted policy, with paired task-success evaluation?
  2. Can you report test-only uncertainty accounting for shared initial states, together with repeated-run variability?
  3. Can you report absolute displacement-tracking gains alongside normalized transfer fractions, or narrow the claim of fully preserved spatial information?
4 comments

We thank the reviewer for an exact and constructive review. Two points led to new measurements and one to a reconsideration of where the paper's centre lies. We answer each with the location of the change (blue text in the revised PDF).

The practical benefit is conditional (W1). We agree. After discussing this review among ourselves, we came to share the reviewer's view that the centre of the paper is not an improvement in success but an analysis of the conditions under which a VLA fails under spatial object perturbation, and we revised the paper in that light. The analysis is this: the displaced position is present in the visual tokens, it is under-used, and a few attention heads route it to the action tokens. The amplification is the causal test of that account, not a remedy, and the title, abstract, §1 and §5 now say so. Where it works, we quantify it (4 to 25% of the clean-to-displaced gap, App. R); where it does not, the same measurements say why (App. M: distributed routing in OpenVLA, direction-dependent writes in GR00T's last DiT block). The narrow factor window stays in the Limitations.

T(0)=1T(0)=1 and absolute tracking (W4, Q3). This point named a weakness of the paper that we had not recognised ourselves, and we thank the reviewer for it; the argument is more rigorous as a result. We agree that T(0)=1T(0)=1 holds by construction and does not establish accurate position information. We removed "fully present", now say the displacement "enters through" the visual rows, and added two absolute measurements. App. L trains a spatial readout on the visual token rows of each layer; at the best layer it recovers kdec≥0.80k_{\mathrm{dec}}\ge0.80 of a 10 cm displacement in all five policies (error 3 to 6 cm, shuffled-label floor 14 to 18 cm). App. N measures the first-closure tracking slope kk of the base policies, 0.20 (OpenVLA-OFT), 0.40 (π0\pi_0), 0.65 (π0.5\pi_{0.5}), 0.36 (GR00T), so use is a fraction of what is decodable (Table L.1). The readout is within-task and a lower bound for the wrist camera (Limitations). The amplification does not move the mean first closure in OpenVLA-OFT and π0\pi_0 (∣Δk∣≤.03|\Delta k|\le.03); the gain comes from recovered episodes whose first closure lands 1.4 to 3.8 cm nearer the object and whose re-grasp loop ends early (App. N, §4.4).

Cluster-aware uncertainty and repeated runs (W3, Q2). The reviewer is right that App. A covered the patching measurements. App. O now treats each initial state as a cluster, with a bootstrap interval and a sign-flip permutation test on per-state sums for every arm (Table O.1). The intra-state correlation of paired differences is .00 (OpenVLA-OFT), .02 (π0\pi_0) and .17 (π0.5\pi_{0.5}), so the correction matters for π0.5\pi_{0.5}, whose pooled gain loses significance (p=.082p=.082). We added 19 held-out states for π0\pi_0 and OpenVLA-OFT. Held-out: π0\pi_0 +8.0+8.0 [+2.7,+13.8][+2.7,+13.8], p=.010p=.010; π0.5\pi_{0.5} +14.7+14.7 [+2.9,+26.0][+2.9,+26.0], p=.039p=.039; OpenVLA-OFT +3.8+3.8 [−0.4,+8.2][-0.4,+8.2], p=.13p=.13, positive in all three held-out estimates but not significant (abstract, §4.4, §5). App. P quantifies run variability: with seeded noise a second base run reproduced all 800 outcomes; without seed sharing, the same episode agrees in 91 to 97% of cases with symmetric flips, so unshared noise adds variance, not bias.

Physical robots and fine-tuned policies (W2, Q1). We share the reviewer's view that a physical robot is where this question must ultimately be settled, and we took a first step in that direction. We ran a preliminary test on a π0\pi_0 policy fine-tuned on a bimanual arm (object moved 20 cm). The clean-patch screen needs a clean scene with the same robot pose and replayed prefix, which we could not reproduce on hardware, so heads were chosen by an open-loop gain probe. Amplifying them removed the old-location failure (20/20 base episodes to 0/15) without raising success significantly (0/20 to 3/15), and a pre-registered specificity test on a second axis failed, as one of four random head sets did as well. Because the heads were not selected by the screen of §3.2, this run does not test the paper's claim, and presenting it would overstate what we have verified; the Limitations therefore say our results are simulation only. We would gladly add it as an appendix if the reviewer wishes. We could test the fine-tuning part. App. W runs the screen on a public π0.5\pi_{0.5} checkpoint fine-tuned on ManiSkill3 (a different robot, camera and objects): it finds the same four heads as on LIBERO (split-half correlation .88), so the heads survive fine-tuning, but amplifying them does not help and produces a constant offset of the first closure. A 20-episode clean-scene check (first-closure shift above 1 cm) flags this setting beforehand; App. V shows it is a necessary no-harm condition across 15 settings, with a post hoc threshold, which we state. §5 names physical validation as the next step.

We thank the reviewer again for a review this specific.

Thank you again for your review. As the discussion period closes on November 18, we would be grateful to know whether our response and the revised PDF address your concerns. Your observation that T(0)=1T(0)=1 holds by construction identified a weakness we had not seen; it led us to measure, in absolute units, how much of the displacement is decodable and how much the policy uses (App. L, N), and to remove "fully present" from the paper. Cluster-aware uncertainty and new held-out states are in App. O and P, and the fine-tuning question is addressed in App. W. If any point remains open, we will respond before the deadline.

Thank you for following up. The narrower claim about positional information and the added decoding and statistical analyses address my concerns about the interpretation of T(0) = 1 and the diagnostic results.

My main remaining concern is whether amplifying the selected attention heads reliably improves real-robot performance. The reported increase from 0/20 to 3/15 successes was not statistically significant, and the random-head control did not establish that the improvement was specific to the selected heads. The revision strengthens the paper as a diagnostic study, but the evidence for a reliable real-robot intervention remains limited. I am therefore keeping my original rating.

Thank you for reading the revision and for confirming that the narrower claim and the added analyses address the T(0)=1T(0)=1 and statistical concerns.

On the real-robot run, we agree that 0/20 to 3/15 does not establish a reliable intervention, and the paper does not claim one. We would add one point about how we read it: success was a poor primary metric on this rig, because the amplification did what the diagnosis predicts and the failure moved elsewhere.

The reach moved. On every episode we measured, from the end-effector trajectory alone, the closest approach to the displaced object and the fraction of low-height steps spent nearer the displaced object than the training-time location (dwell). At 20 cm toward −x-x, the base policy reached within 11.7 cm with a mean dwell of 0.4% (19 of 20 episodes at 0%); with the candidate heads amplified by 2 it reached within 2.7 cm with a mean dwell of 91% (14 of 15 episodes above one half); a random set at the same factor reached within 11.2 cm with a mean dwell of 34% (4 episodes to the object, 6 at 0%). position prior is removed on the robot as in simulation, and on this axis the random-head control separates from the selected heads.

The grasp then fails for a reason the heads do not control. The intervention converted old-location failures into failures at the displaced object (10 of 15 candidate episodes). Success on this rig also requires the posture of the gripper, and its tilt rose from 3.6° in clean scenes to 16 to 18° under the displacement in every arm, base and intervened alike, so the gripper knocked the bottle over or closed beside it. A screen of all 288 heads found that tilt is not carried by the heads that carry position (rank correlation .02). One correction to our earlier reply: one candidate episode closed at the training-time location, so that count is 1 of 15, not 0.

How the heads were chosen. The clean-patch screen of §3.2 needs paired clean and displaced observations with the same robot pose and a replayed prefix, which cannot be produced on hardware. The heads were therefore chosen by an open-loop probe on stored observations; Appendix X states this and the other differences from the simulation study in its first paragraph. The limits of the run are stated in Appendix X.

The run is in the revised PDF as Appendix X (Tables X.1 to X.3, Figure X.2), with two episode replays in the supplementary material, and the Limitations in §5 point to it. We read it as consistent with the diagnosis: the heads carry the position and amplifying them moves the reach, and the remaining failure lies in a component they do not control.

Thank you again for the careful reading.

Reviewer_qbRNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When the target object is displaced, all five policies reach toward its previous location. The paper asks whether the new location is absent from the network or simply unused, and settles it with activation patching: restoring the visual token rows to their clean-scene values removes the response, so the information was present throughout. A head-level screen then identifies a small number of attention heads that carry the displacement into the action tokens, and scaling those heads' input-dependent deviation — mean preserved, no training — recovers object-following behaviour in several policies. The control set is the strongest part of the work: random heads, inert heads matched on deviation magnitude, and heads matched on total injected change are all ineffective.

Critical strengths
  1. Rather than proposing another remedy, the paper asks whether the new location is absent from the network or present but unused. A transfer fraction of one at zero displacement, together with the observed failure to reach for the old location, establishes non-use rather than absence — a cleaner causal statement than most VLA interpretability papers manage, and one that changes how the failure should be understood.
  2. The phenomenon — several independent policies all reaching for a target's previous location after it is moved — is universal across the evaluated models, which makes the diagnosis broadly relevant rather than specific to one architecture.
  3. The intervention is training-free and applied to a frozen backbone, so it inherits the base policy everywhere the displacement response is not involved.
  4. The method unfolds in a locate–screen–intervene–control order, notation matches the figures, and both success and failure cases receive a mechanistic explanation rather than being reported as outcomes.
Critical weaknesses
  1. A single perturbation axis; the evidence base for generality is narrow. Simulation only, a handful of checkpoints, and a single perturbation axis. The claim that spatial-robustness failure is a routing problem rests on planar target displacement alone; viewpoint shifts, camera perturbation, and scene clutter are untested.
  2. The intervention is applied several layers downstream of the damage and closes only a small fraction of the gap. Comparing clean success to displaced success and to the intervened result, the intervention recovers only a small fraction of the gap — the large majority of the failure remains, and that fraction is never stated. A contemporaneous encoder-side modulation result, using very few parameters and attributed to representation drift in the visual encoder, recovers far more on a related axis and is not cited. If that diagnosis is right, the intervention here is applied several layers downstream of where the damage occurs.
  3. The abstract reports pooled values while the conclusion argues from held-out values — the two are inconsistent. The head set, the number of heads, the gain, and the subspace variant are all selected on validation, so only the test condition is unbiased. On the held-out column the gains are smaller for some policies and one loses its significance marker entirely; for one policy both the pooled and held-out numbers are non-significant while its matched control is significantly harmful. The net result is that fewer policies reach significance on the held-out column than the conclusion states.
  4. The screening cost is not reported. The screening cost is one forward pass per initial state × grid cell × head, and no wall-clock or GPU cost for screening is reported — which matters for assessing whether the method is usable outside the study.
Questions for authors
  1. Can the abstract report the held-out test column, or label both columns clearly? Do you agree that fewer policies reach significance on the test condition than the conclusion states?
  2. What fraction of the clean-to-displaced gap does the intervention close, per policy?
  3. Will the full displacement-magnitude sweep be shown per policy rather than a per-policy selected magnitude, and will the displacement-subspace variant be reported for all policies?
2 comments

We thank the reviewer for a careful reading and for stating the causal question in a form ("absent or present but unused") that we have adopted in the revision. We answer each point and give the location of the change (changed text is blue in the revised PDF).

Pooled versus held-out values (W3, Q1). We agree on both counts. Only the held-out states give an unbiased estimate, and fewer policies reach significance there than the submitted conclusion stated. We added a cluster bootstrap interval and a sign-flip permutation test over initial states for every arm (App. O) and 19 further held-out states for π0\pi_0 and OpenVLA-OFT. Held-out: π0\pi_0 +8.0+8.0 [+2.7,+13.8][+2.7,+13.8], p=.010p=.010; π0.5\pi_{0.5} +14.7+14.7 [+2.9,+26.0][+2.9,+26.0], p=.039p=.039; OpenVLA-OFT +3.8+3.8 [−0.4,+8.2][-0.4,+8.2], p=.13p=.13, positive in each of three held-out estimates but not significant. The abstract, §4.4 and §5 now report these numbers, "three of five" is removed, and we state that the pooled gain of π0.5\pi_{0.5} loses significance under clustering (p=.082p=.082). For OpenVLA, Table 1 shows what the reviewer notes: the matched control is significantly harmful while the amplification is not significant in either direction.

Fraction of the gap closed, and the encoder-side result (W2, Q2). The fraction was indeed missing. App. R (Table R.1) gives it per policy: 4 to 25% on LIBERO-Object, 41 to 59% on the LIBERO-Pro conditions where the base policy is off the floor. To place these values we trained a reference adapter for OpenVLA-OFT with the vision encoder frozen (rank-32 LoRA on the backbone output projections): it closes 45% of the gap on validation and test states, and a LoRA on all backbone linear layers closes 59%. For this axis, therefore, the encoder output carries enough of the displacement for a downstream readout to close about half of the gap, which locates the shortfall downstream of the encoder. We believe the encoder-side work the reviewer refers to is Li et al. (2025, arXiv 2512.02902), which modulates visual tokens for viewpoint shifts; it is now cited in §2 and contrasted as the complementary case. If the reviewer meant a different paper, we would be grateful for the reference.

After reading this review we also changed the framing of the paper. The contribution is the diagnosis: the displaced position is present (decodable to within a few centimetres, App. L), it is under-used (the first grasp follows 20 to 65% of it, App. N), and a few heads route it. The amplification is the causal test of that account, and §4.4 and §5 now say explicitly that it closes only part of the gap. The title was changed accordingly ("Diagnosing ... with Attention-Head Patching and Amplification").

A single perturbation axis (W1). We agree that the evidence for generality is narrow and now list this first in the Limitations. We chose one axis deliberately: a planar displacement with paired clean and moved scenes is a single controlled variable, and that control is what allows the patching, the screen and the intervention to act on the same quantity and support a causal statement. We also agree that the question of other axes must be asked, so we ran three experiments. (i) Axis specificity (App. S): on OpenVLA-OFT the screened heads carry at most 3% of the plan response to a camera translation or rotation, which a disjoint set of heads in layers 19 to 30 carries; in π0\pi_0 the top head also responds when a rotation shifts the object in the image, so the heads respond to the object's image position rather than to the task. (ii) Non-target objects (App. T): the amplification is not target-selective (OpenVLA-OFT loses 6.4 points when only a non-target moves). (iii) The other LIBERO-Plus families (App. U): neutral for OpenVLA-OFT and π0\pi_0, −2.6-2.6 points for π0.5\pi_{0.5}. These results are summarised in a new paragraph "Scope of the lever" in §4.4.

Screening cost (W4). The cost was reported in App. K (Table K.1: 2.8 GPU-hours for the 1,024 heads of OpenVLA-OFT, about 1.5 for π0\pi_0, π0.5\pi_{0.5} and GR00T, under 1 hour for the attribution approximation, each on one A100), but we realise it was easy to miss there.

Magnitude sweep and subspace variant (Q3). App. Q gives the paired net change at every magnitude run (5 to 20 cm) for all five policies, with cluster intervals, and §3.4 now states the rule that fixed the reported magnitude (the largest at which base success lies between 10 and 90%, chosen before any intervention). Table Q.1 now also covers OpenVLA at 10 cm: base 0.0%, +3.0+3.0 points pooled (p=.008p=.008) but +1.5+1.5 and not significant on held-out states. The subspace variant is now reported for every policy (App. I): it does not create a lever where the full-deviation variant has none (OpenVLA: clean success 75→40%75\to40\% at rank 4, factor 2) and the variance fractions are given at a common rank.

We hope these changes answer the concerns, and we thank the reviewer again for comments that improved the paper substantially.

Thank you again for your review. As the discussion period closes on November 18, we would be grateful to know whether our response and the revised PDF address your concerns. Your point that the abstract reported pooled values while the conclusion argued from held-out values led us to report held-out, cluster-aware numbers throughout (Abstract, §4.4, §5, App. O), and your question about the fraction of the gap closed is now answered in App. R (4 to 25%, against 45 to 59% for a LoRA reference). The full magnitude sweep (App. Q, now including OpenVLA at 10 cm) and the subspace variant for all policies (App. I) are also in the revision. If any point remains open, we will respond before the deadline.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers. The revised PDF integrates the experiments run during the rebuttal. We list what changed, where, and which comment it answers.

1. Framing: an analysis, with amplification as a causal test (Title, Abstract, §1, §5; mkZ7, iEyi). The title now reads "Diagnosing the Robustness of VLAs to Spatial Object Perturbation with Attention-Head Patching and Amplification". The contribution is stated as the diagnosis (the displaced position is present but under-used, and a few heads route it); the amplification tests that diagnosis and recovers part of the lost success. All headline numbers are now held-out instead of pooled: π0\pi_0 26.4→34.5%26.4\to34.5\% and π0.5\pi_{0.5} 41.3→56.0%41.3\to56.0\% (Abstract, §4.4, §5), OpenVLA-OFT 9.6→13.4%9.6\to13.4\%, not significant (§4.4, §5). "Three of five policies" became "two significant on held-out states, one consistent but not significant".

2. Cluster-aware statistics and new held-out states (§4.4, Table 1 caption, App. O, P; mkZ7, prrx, qbRN). Initial states are treated as clusters, with a cluster bootstrap interval and a sign-flip permutation test per arm (Table O.1). We added 19 held-out states (new seeds) for π0\pi_0 and OpenVLA-OFT. Held-out results: π0\pi_0 +8.0+8.0 [+2.7,+13.8][+2.7,+13.8], p=.010p=.010; π0.5\pi_{0.5} +14.7+14.7 [+2.9,+26.0][+2.9,+26.0], p=.039p=.039; OpenVLA-OFT +3.8+3.8 [−0.4,+8.2][-0.4,+8.2], p=.13p=.13. The pooled gain of π0.5\pi_{0.5} is no longer significant (p=.082p=.082). App. P quantifies run-to-run variability.

3. Presence versus use, in absolute units (§4.2, §4.3, Fig. 3 caption, App. E, L, M, N; mkZ7). "Fully present" is removed, and T(0)=1T(0)=1 is described as holding by construction. A spatial readout on the visual tokens recovers kdec≥0.80k_{\mathrm{dec}}\ge0.80 of a 10 cm displacement in all five policies (error 3–6 cm), while the first gripper closure follows k=0.20k=0.20–0.650.65 of it (Table L.1). App. N shows that amplification does not move the mean first closure in OpenVLA-OFT and π0\pi_0 (∣Δk∣≤.03|\Delta k|\le.03); the gain comes from recovered episodes (first closure 1.4–3.8 cm closer, 7–14 closures reduced to 3–5). App. M gives the head-level picture (attention of the two reader heads of OpenVLA-OFT on the object's wrist patches: .34.34/.33.33 in the clean scene, .15.15/.19.19 after the move).

4. Effect size and location of the shortfall (§2, §4.4, App. R; qbRN). The amplification closes 4–25% of the clean-to-displaced gap (Table R.1, Fig. R.1). For reference, a LoRA on the language backbone with the vision encoder frozen closes 45% (59% on all linear layers), so for this axis the shortfall lies downstream of the encoder. The viewpoint study of Li et al. (2025) is now cited and contrasted.

5. Why OpenVLA and GR00T have no lever (§4.4, App. I, M, V; iEyi, prrx). Decodability matches the other three policies, so the cause is not the representation. In OpenVLA the routing is distributed (cosine of head write with residual change .08.08–.32.32, against .72.72 in OpenVLA-OFT). In GR00T the heads of the last DiT block write 3–4 times more toward −x-x. A clean-scene check (first-closure shift ≤1\le1 cm on 20 episodes) is a necessary no-harm condition (0 of 8 settings harmed below it, 4 of 7 above), not a predictor of gain, and its threshold is post hoc (Limitations).

6. Scope of the lever (new paragraph in §4.4, App. Q, S, T, U, W; qbRN, iEyi, prrx). Axis specificity: the heads carry ≤3%\le3\% of the response to a camera translation or rotation in OpenVLA-OFT; in π0\pi_0 the top head also responds when a rotation shifts the object in the image. Non-target objects: the amplification is not target-selective (OpenVLA-OFT −6.4-6.4 points when only a non-target moves; π0\pi_0 +9.0+9.0 and π0.5\pi_{0.5} +12.3+12.3 when both move). Other LIBERO-Plus families: neutral for OpenVLA-OFT and π0\pi_0, −2.6-2.6 for π0.5\pi_{0.5}. Transfer without re-screening to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 with the same checkpoints (App. V): π0.5\pi_{0.5} +10.6+10.6 on LIBERO-Goal (p=.005p=.005), the other five settings neutral. Full 5–20 cm sweep, now with OpenVLA at 10 cm, under the magnitude-selection rule (§3.4, Table Q.1). On a ManiSkill3 fine-tuned π0.5\pi_{0.5} the screen finds the same four heads, but the amplification does not transfer (App. W).

7. Method clarifications and corrections (§3.2, §3.4, Table 1, App. A, B, I). The screening cost is in App. K (1.5–2.8 GPU-hours). §3.4 states what differs from ActAdd and ITI. Corrections: heads are ranked by ghg_h, not by ghg_h over its standard error; Gh>1G_h>1 for π0\pi_0 is a normalization artifact (transfer fractions .35.35 and .11.11); subspace variance fractions are now all at rank 4.

8. Limitations (§5) now list one perturbation axis, simulation only, a partial lever, no significant held-out gain in OpenVLA-OFT, non-selectivity for the target, interference in π0.5\pi_{0.5}, a within-task readout (a lower bound for the wrist camera) and the post hoc threshold.

Dear Reviewers,

Thank you again for your reviews. On November 11 we posted a revised PDF (changed text in blue; Appendices L–W are new), the summary of the revision above, and a response to each review. As the discussion period closes on November 18, we would be grateful if you could let us know whether our responses address your concerns, and whether any point remains open that we can still clarify before the deadline.

Several of your comments changed the paper substantively. Reviewer mkZ7's observation that T(0)=1T(0)=1 holds by construction led us to measure, in absolute units, how much of the displacement is decodable and how much is used (App. L, N). Reviewer qbRN's point that the abstract reported pooled values while the conclusion argued from held-out values led us to report held-out, cluster-aware numbers throughout (App. O) and to state the fraction of the gap the intervention closes (App. R). Reviewer iEyi's question about what distinguishes the intervention from activation steering led to a precise statement in §3.4 and to a fixed rule for the reported magnitude (App. Q). Reviewer prrx's question about multiple objects and unseen scenes led to the non-target experiments (App. T) and the transfer to LIBERO-Spatial, LIBERO-Goal and LIBERO-10 (App. V).

On the two concerns several of you raised, generality across policies and the single perturbation axis, we have tried to answer with measurements rather than argument: the revision quantifies where the lever works and why it does not elsewhere (App. M, V, W), tests other perturbation axes (App. S, U), and restates the contribution as the diagnosis of a failure under one controlled variable, with the amplification as its causal test. Both concerns are stated as limitations in §5.

We would welcome any further questions and will respond promptly.

The Authors

We thank the four reviewers for reviews specific enough to turn into experiments. The revised PDF (changed text in blue; Appendices L–X are new; main text 10 pages) and the supplementary material (LIBERO videos of Figure I.1, two robot replays of Appendix X) are uploaded. We list the strengths the reviewers noted, the main points raised, and what changed.

Strengths noted in the reviews

  1. The question is whether the new location is absent from the network or present but unused, and the activation-patching evidence separates the two; this is a cleaner causal statement than documenting the failure (qbRN, iEyi, prrx).
  2. The failure is shared by five policies with different action-generation mechanisms, so the diagnosis is not specific to one architecture (qbRN, iEyi).
  3. The same head outputs serve the screen and the intervention, which is training-free on a frozen backbone and inherits the base policy wherever the displacement response is not involved (mkZ7, qbRN, prrx).
  4. The evaluation is controlled: paired outcomes, separate validation and held-out states, random, deviation-matched and energy-matched controls, transfer to LIBERO-Pro, and a mechanistic account of both the successes and the failures (mkZ7, prrx, qbRN).

Main points raised and how we addressed them

  1. The amplification reads as a remedy, and its benefit is conditional (mkZ7, iEyi). We reframed the paper as a diagnosis of the failure, with the amplification as its causal test rather than a fix; the title, abstract, §1 and §5 say so, and App. R states that the lever closes 4 to 25% of the clean-to-displaced gap.
  2. T(0)=1T(0)=1 holds by construction and does not show that accurate position information is present (mkZ7). We removed "fully present" and added absolute measurements: a spatial readout recovers at least 80% of a 10 cm displacement from the visual tokens (App. L), while the first gripper closure follows only 20 to 65% of it (App. N).
  3. The abstract used pooled numbers, episodes sharing an initial state were treated as independent, and run-to-run variability was not reported (qbRN, mkZ7, prrx). All headline numbers are now held-out, every arm has a cluster bootstrap interval and a permutation test over initial states, 19 held-out states were added, and run variability is quantified (App. O, P).
  4. Two of the five policies gain nothing, and the conditions for success are not characterised (iEyi, prrx). App. M shows that the position is decodable in all five policies and that the difference lies in how the heads route it (distributed in OpenVLA, direction-dependent in GR00T); App. V gives a 20-episode clean-scene check that separates harmful from harmless settings.
  5. A single perturbation axis; viewpoint, clutter, multiple objects and other tasks are untested (qbRN, iEyi, prrx). We tested camera translation and rotation (App. S), moving non-target objects (App. T), the other LIBERO-Plus families (App. U), and transfer to LIBERO-Spatial, Goal and 10 without re-screening (App. V); the results bound the scope of the lever and are stated as limitations.
  6. The full magnitude sweep, the subspace variant, the screening cost, and the difference from activation steering should be reported (qbRN, iEyi). These are in App. Q, App. I, App. K and §3.4.
  7. Validation on a physical robot with a hardware-adapted policy (mkZ7, prrx). New in this revision, App. X reports a preliminary run on an ALOHA robot with a fine-tuned π0\pi_0: the robot shows the same failure mode as the simulated policies, closing at the training-time location in every base episode, and amplifying four displacement-carrying heads moves the gripper to the displaced object (closest approach 11.7 to 2.7 cm). Because a clean-scene replay is not available on hardware, the heads were selected with an open-loop variant of the screen, which the appendix describes.

We thank all the reviewers again for taking part in the review and the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.