acceptodds
Under review as a conference paper at ICLR 2027

Observing and Steering Affordance in VLA Hidden States for Spatial Robustness

Abstract

Vision-language-action (VLA) policies achieve strong performance on manipulation benchmarks, but their success rate degrades sharply once object positions drift outside the training distribution, limiting real-world deployment. We ask whether a VLA's internal representations already encode a high-level semantic feature that reflects task progress, and whether this feature can be observed and controlled at inference time, without retraining the policy. We show that a linear probe trained on language-backbone hidden states reliably decodes an affordance-based cost derived from 3D value maps, and that this cost can be driven toward a target value at test time through targeted activation perturbations along the probe's gradient direction. A layer-wise causal analysis identifies intervention layers where this perturbation both lowers the predicted cost and changes the resulting action, which we use to select where steering is applied for each task. We evaluate this observe-and-control approach on two recent open-source VLA backbones, OpenVLA and , under a simulated closed-loop out-of-distribution benchmark built on LIBERO-Pro, in which the target object is displaced by an increasing radius from its training position. Without modifying any policy weights, steering raises out-of-distribution task success from 16.6% to 29.5% on OpenVLA and from 34.5% to 41.1% on , an average absolute improvement of nearly 10 percentage points across both backbones.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Representation Steering, Robot Manipulation, Out-of-Distribution Generalization, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h2WyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an observe-and-steer framework to improve the spatial robustness of vision-language-action (VLA) policies without updating their weights. The method trains linear probes to decode a 3D affordance cost from VLA hidden states and uses probe-derived directions to steer internal activations at inference time. It further introduces intervention gating, layer selection, and adaptive strength selection through bisection. Experiments on LIBERO-Object with displaced target objects demonstrate improvements in task success on both OpenVLA (16.6% to 29.5%) and π0.5 (34.5% to 41.1%).

Critical strengths
  1. Meaningful connection between representation analysis and control. Rather than merely demonstrating high probing accuracy, the paper uses affordance costs decoded from VLA hidden states to guide actual robot behavior. This connection between representation analysis and closed-loop control is an interesting contribution.

  2. Promising experimental improvements. The method improves task success from 16.6% to 29.5% on OpenVLA and from 34.5% to 41.1% on π0.5, with matched evaluation scenarios within each backbone. These results demonstrate practical benefits across two different VLA architectures.

Critical weaknesses
  1. Insufficient evidence for affordance-specific steering. Reducing the probe's predicted cost does not necessarily imply that the resulting actions are better. Although the paper shows that interventions change actions, it does not establish that the probe-derived direction is more effective than generic perturbations. Key controls, such as norm-matched random directions and simpler target-distance probes, are missing. This weakens the central claim that the method specifically controls affordance representations.

  2. Unclear layer selection and missing critical ablations. The appendix describes how candidate intervention layers are identified, but the final layer choices and selection procedure are not sufficiently documented, particularly whether selection is independent of the OOD test cases. Moreover, the paper does not evaluate probe accuracy under object displacement or isolate the benefits of adaptive bisection over fixed intervention strengths. These omissions make it difficult to attribute the reported improvements to the proposed mechanisms.

Questions for authors
  1. Layer selection (clarification). Which intervention layer was ultimately selected for each task? What data and criteria were used for selection, and was the process independent of the OOD evaluation scenes?

  2. Steering effectiveness (additional experiments). How does the method perform with norm-matched random perturbation directions or a probe trained using only target distance instead of the full affordance cost field? How much improvement does adaptive bisection provide over a fixed intervention strength?

  3. OOD probe validity and behavioral effects (additional analysis). How accurately does the probe predict affordance cost after target objects are displaced? After steering, does the actual geometric affordance cost decrease over subsequent executed steps, rather than only the probe's predicted cost decreasing immediately after intervention?

1 comment

Thanks to the authors for the detailed rebuttal and additional experiments. Some of my concerns have been addressed, especially the fixed-strength comparison and OOD probe evaluation.

However, I am still concerned about the layer selection, as the authors confirmed that OOD evaluation scenes were used to select the intervention layers. This makes it difficult to judge how well the method generalizes to unseen test cases.

Also, the improvement over random directions on π0.5 is quite small (65.0% vs. 63.3%), and the full affordance cost only slightly outperforms target distance supervision (25.0% vs. 21.7%). These results still leave some doubts about whether the improvement really comes from affordance-specific steering.

Overall, I appreciate the additional efforts, but I will keep my original rating.

Reviewer_P1tgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an observe-and-steer framework to improve the spatial robustness of vision-language-action (VLA) models without updating the underlying policy weights. The method trains linear probes to predict a 3D affordance cost from VLA hidden states and uses probe-derived directions to steer intermediate activations during inference. A layer-wise intervention analysis guides the selection of effective intervention layers, while adaptive gating and bisection determine when and how strongly to intervene. Experiments on OpenVLA and π0.5 demonstrate improved task success under object-position perturbations in LIBERO-Object.

Critical strengths
  1. The paper explores an interesting direction by connecting interpretability of VLA hidden representations with inference-time robot control.
  2. The proposed method does not require fine-tuning the underlying VLA policy, making it potentially applicable to pretrained models.
  3. The evaluation includes two different VLA architectures, and the layer-wise analysis provides useful observations about the relationship between predicted affordance cost and action changes.
Critical weaknesses
  1. The linear probes achieve high prediction accuracy on held-out in-distribution trajectories, but their accuracy under object displacement is not evaluated. Since the affordance cost is largely determined by the end-effector position relative to the target, and target locations remain within the nominal training distribution during probe training, the probe may primarily learn position- or trajectory-related correlations rather than a robust representation of object affordance. It remains unclear whether the probe can accurately reflect changes in target affordance when the object is displaced.
  2. The steering direction and perturbation magnitude are selected to decrease the cost predicted by the same linear probe. However, reducing the probe's output does not necessarily imply that the resulting robot action moves toward a genuinely better physical state. The layer-wise analysis already shows that certain interventions can substantially reduce the predicted cost without meaningfully changing the emitted action. Although the final success-rate improvements are encouraging, a more direct analysis of actual affordance-cost changes and action correctness would strengthen the causal interpretation.
  3. The main experiments compare the proposed method primarily against the original VLA policies without steering. However, the paper lacks comparisons with other inference-time correction approaches, including related affordance-based intervention and probe-guided recovery methods. And the experiments are conducted only in simulation, but not in the real-world senarios.
Questions for authors
  1. Can the authors evaluate probe prediction accuracy on the OOD displaced-object trajectories? How does it compare with a simpler probe based on end-effector position or proprioceptive state?
  2. Can the authors measure whether steering actually reduces the ground-truth 3D affordance cost or improves end-effector alignment with the target, rather than only decreasing the probe's predicted cost?
  3. What is the actual inference overhead introduced by probe evaluation and repeated forward passes during bisection?
  4. Some experiments in the real-world robot platform will strenthen the paper.
Reviewer_2dWyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an inference-time activation steering framework to improve the spatial robustness of VLA models. The method trains linear probes to predict affordance costs from VLA hidden states and uses the learned directions to modify hidden representations toward lower-cost states. Experiments on LIBERO-Object demonstrate improved success rates for both OpenVLA and π0.5\pi_{0.5} under object position perturbations.

Critical strengths
  1. The method leverages affordance information from VLA hidden states to correct robot actions at inference time, without requiring policy fine-tuning.

  2. The proposed method consistently improves the spatial robustness of different VLA backbones under object position perturbations.

  3. The layer-wise analysis in Figure 4 provides useful insights into how hidden-state interventions affect both predicted affordance costs and the resulting robot actions.

Critical weaknesses
  1. The experiments are restricted to object position perturbations in LIBERO-Object. Evaluations on additional benchmarks, such as LIBERO-Plus or RoboTwin, would help demonstrate the generalization of the proposed method to more diverse tasks and distribution shifts.

  2. The proposed affordance cost is mainly based on target distance and obstacle avoidance. It remains unclear whether this formulation can effectively guide more complex manipulation tasks that require precise contact reasoning or multi-stage interactions.

Questions for authors
  1. Since the method relies on affordance costs, how does the proposed hidden-state steering compare with a simpler approach that directly uses affordance costs to guide VLA actions during inference?

  2. The linear probes are trained and validated on in-distribution trajectories. How accurate are the predicted affordance costs under OOD conditions, especially as object displacement increases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.