FIPAS: Factor-Invariant and Position-Aware Activation Steering for Robust VLA Models

Vision-language-action (VLA) policies can lose closed-loop reliability under visual changes that preserve the intended task. Post-hoc activation steering can adapt a frozen policy, but success–failure contrasts may entangle environmental variation with action error and task progress. We introduce FIPAS, which estimates low- rank environment-correlated subspaces from behavior-matched clean–perturbed activations, protects an estimated task-predictive subspace, and conditionally cor- rects abnormal residuals at calibrated layer–action-query sites. A gate based on current or past rollout signals attenuates intervention during sensitive manipulation stages. In a 50-task RoboTwin aggregate, FIPAS raises randomized success from 20.88% to 36.90% on the same clean500 policy, while clean success changes from 85.40% to 86.34%. In a 10-task factor-controlled ManiSkill3 aggregate, gains under isolated and composed shifts range from 5.83 to 9.40 percentage points, with a −0.21-point clean change. These reported aggregates support selective hidden-state correction as a complement to clean-data scaling under calibrated visual shifts.