acceptodds
Withdrawn from ICLR 2027

Sparse Feature Analysis and Control of Vision-Language-Action Models under Visual Perturbations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models have demonstrated strong manipulation performance, yet often degrade under visual perturbations such as camera, background-texture, lighting, and sensor-noise shifts. We ask which internal features make a VLA visually sensitive, what these features represent, and whether intervening on them produces the predicted change in behavior. To obtain localized analysis units, we decompose raw VLA activations using sparse autoencoders (SAEs), and introduce a shared-state paired perturbation protocol that isolates feature responses to visual shifts while controlling for physical state divergence. We then identify compensator features, which serve as action-coupled steering targets, and detector features, which predict perturbation-induced failures. Across autoregressive and diffusion-based VLA architectures, sparse features provide more localized perturbation-relevant units than raw residual dimensions, and the two roles form disjoint populations: detectors in the vision-language stream or early layers and compensators in the action expert or late layers. By relating features to the policy’s own action modes, we find that compensators encode task-invariant motion primitives and that a perturbation weakens the policy’s commitment to the primitive it is executing rather than switching it to another. Finally, we verify the interpretation by intervention: steering a single compensator feature shifts perturbed actions back toward the unperturbed action and modestly improves rollout success, and gating it with a detector makes it perturbation-specific. Together, these results identify, interpret, and verify the features behind visual sensitivity in VLAs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action, Mechanistic Interpretability, Sparse Autoencoders, Visual Perturbations, Feature Steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xY5AOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains sparse autoencoders (SAEs) on two open source frontier VLA models: OpenVLA-oft and Pi05. A common phenomenon when VLA models are fine-tuned on small non-diverse dataset is a loss of generality [LIBERO-PRO Xueyang Zhou 2025]. The authors investigate this by comparing the activations of features across clean episodes and perturbed episodes (perturbed covering camera, background, light and sensor). They find that certain features can reliably detect the presence of perturbations, terming these detector features. They also find another class of feature which they term compensator features that by steering can improve success rate on perturbed episodes.

Critical strengths

Understanding the mechanisms behind VLA brittleness is an incredibly important pursuit. I find the paper to be well motivated, and the abstract and introduction to be clear and educational.

The paper includes code and experimental outline in the appendix sufficient to replicate this work.

Critical weaknesses

No strong baseline to the detector features is presented. If the authors are claiming that SAE features can detect perturbation signals they should compare against competent baselines like linear probes. The raw residual baseline is not a fair comparison.

The improvements of steering features on task success rate is very small and virtually identical to the random SAE feature baseline. 79.0% success rate unsteered versus 79.9% steered. Which is less than 1%. (Table 4.). It is also unclear how many trials each success rate in the table represents. At the top of the section it says that “All experimental results are averaged over three trials“, does this mean three trials across every task in the LIBERO-Plus dataset ? Again Table 6 shows small, but consistent changes in success rate. Including the number of trials and confidence intervals would strengthen the paper. I understand this information may be in the appendix, but it is difficult to parse all 20 pages for this review.

I have some fundamental concerns about the selection of the compensator features. Since they are all evaluated in closed loop on the same set of tasks before being chosen, it is difficult for them not to improve slightly on the base policy. The authors state that they select and test the top .1% of features, which could be hundreds of features. Even if the true effect is zero, selecting the best feature across N candidates with some random noise we would expect to see a slight improvement over the base policy. To alleviate these concerns the episodes / tasks that these features are calibrated on need to be different from the episodes/tasks that the success rate is evaluated on.

Results are limited in scope as they only apply to a single, often overfit [LIBERO-PRO Xueyang Zhou 2025] simulation environment. It would significantly strengthen the paper to include hardware both in the small dataset scale (to replicate the sort of overfitting which is occurs in LIBERO) and then additionally on a large scale dataset such as DROID (for which there are available FTs of Pi and OpenVLA).

Questions for authors

The compensator features are chosen by closed-loop screening over potentially hundreds of features. Are the tasks used to screen these features disjoint from those used to compute the success rates in Table 4 and Table 6 ?

Why is no baseline against linear probes or individual MLP neurons included for the detector features ? The raw residual baseline seems quite weak and not comparable to SAE features.

Reviewer_Y5qTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes how visual perturbations affect VLA models using sparse autoencoders and a shared-state paired perturbation protocol. Experiments on OpenVLA-OFT and Pi0.5 identify detector features associated with failures and compensator features coupled to action changes. Compensators are interpreted through action modes, and feature steering, including detector-conditioned interventions, is evaluated in simulation, showing modest improvements in closed-loop task success.

Critical strengths
  • The shared-state paired protocol compares feature and action changes induced by unperturbed and perturbed observations while holding robot and object configurations fixed. This reduces confounding from trajectory divergence and helps isolate the effects of visual perturbations.

  • By combining feature analysis, fixed-state interventions, and closed-loop execution, the authors demonstrate the behavioral effects of selected features and modest improvements in task success under perturbations across two VLA models.

Critical weaknesses
  • The absence of physical-robot validation is a major limitation for this paper. Feature discovery relies on controlled shared-state pairing, yet reliable feature selection, gate calibration, and effective steering on real robots have not been demonstrated. Modest gains on two models within the LIBERO/LIBERO-Plus benchmark family provide insufficient evidence of the method’s practical applicability.

  • The evaluation protocol for Tables 4 and 6 remains unclear, including test instances, episode counts, baseline pairing, and separation between feature selection and evaluation. Moreover, offline detection uses paired feature shifts, whereas online gating relies on current activations; the reported detection metrics may therefore not reflect the actual online gate.

  • Zero overlap between small candidate sets selected using different criteria provides limited evidence for stable functional separation without an appropriate overlap baseline. Averaging action chunks also discards temporal structure, leaving the motion-primitive interpretation dependent on an insufficiently validated representation choice. Comparisons using different compensators further prevent isolating the benefit of gating.

  • Several inconsistencies require clarification. FVU < 0.15 is presented as a retention criterion in Section 3.1 but a reference target in Appendix B; commitment decreases are reported as 70%-90% in Section 5 but 63%-80% in Table 23; and L01F3670 has an AUROC of 0.876 in Table 2 but 0.777 in Table 12a.

Questions for authors
  • Figure 1 should be reorganized as it is difficult to get what the exact meaning.
Reviewer_14iiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how visual perturbations affect the representations and behavior of vision-language-action policies. It uses sparse autoencoders and a shared-state protocol that compares clean and perturbed observations at identical simulator states. The analysis identifies failure-predictive detector features and action-coupled compensator features, relates compensators to recurring action modes, and evaluates direct and detector-conditioned feature steering. Experiments on OpenVLA-OFT and π0.5 across four visual perturbation families in LIBERO-Plus include feature comparisons, fixed-state intervention tests, and rollout evaluation. The results suggest that selected sparse features provide useful units for analyzing visual sensitivity and can modestly improve perturbed-policy success.

Critical strengths
  1. The shared-state protocol provides a well-controlled way to study visual sensitivity in robot policies. Section 3.2 compares clean and perturbed observations while holding the underlying robot and object configurations fixed, avoiding the confound of trajectory divergence. Applying this protocol to OpenVLA-OFT and π0.5 across camera, lighting, background, and sensor perturbations provides useful evidence across different policy architectures and visual shifts. This design strengthens the attribution of observed feature and action changes to the visual perturbation.

  2. The behavioral interventions provide useful evidence beyond activation correlations. Tables 4 and 5 compare selected compensators with random SAE features, raw residual dimensions, and other sparse features, examining both rollout success and immediate action changes. For OpenVLA-OFT under camera shifts, Table 5 reports that steering the selected feature moves actions toward the unperturbed action in 70% of tested states, compared with 24-50% for other features. These controls support the usefulness of the discovered features as intervention directions.

Critical weaknesses
  1. The central interpretation of compensators as task-invariant motion primitives is not sufficiently distinguished from alternative action-related representations. Section 5 reports consistent action-mode selectivity across tasks, and Table 5 demonstrates that selected features can influence actions in the predicted direction. However, these findings do not exclude features that represent perceived geometry or task state consistently associated with the same action across tasks. Section 5 also acknowledges that many other features are highly mode-selective, while the intervention controls are not shown to be matched for selectivity to the same mode. This leaves the paper’s central mechanistic interpretation less firmly established than its evidence for useful steering.

  2. The experiments do not adequately isolate the contribution of detector gating. Section 6.2 attributes improved recovery and reduced collateral degradation to gating, but most comparisons between Tables 4 and 6 also change the compensator feature. For example, OpenVLA-OFT camera steering uses L23F4722 in Table 4 and L30F5719 in Table 6, so the success increase from 50.7% to 52.5% cannot be attributed specifically to gating. A comparison holding the compensator and steering strength fixed, with separate measurements of recovered failures and newly introduced failures, is needed to establish the detector’s added value. This limits the evidence supporting a central component of the proposed intervention framework.

Questions for authors
  1. Do the selected compensators outperform other features matched for selectivity to the same action mode and comparable steering strength when measuring correction toward the unperturbed action at shared states?

  2. With the compensator, steering strength, and evaluation initial states held fixed, how do detector-gated and always-on steering compare in recovering Broken rollouts and preserving Resistant rollouts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.