PRESERVE WHAT MATTERS: COUNTERFACTUAL TASK-VECTOR DECOMPOSITION FOR REPLAY-FREE CONTINUAL VISION-LANGUAGE-ACTION LEARNING

Vision-Language-Action (VLA) models have demonstrated strong capabilities in language-conditioned robotic manipulation. Yet operating in open-ended environ- ments requires continual acquisition of new skills, which can severely overwrite previously learned behaviors and lead to catastrophic forgetting. Existing replay- based methods mitigate forgetting by revisiting stored demonstrations, incurring persistent storage and repeated training costs, while replay-free approaches typi- cally preserve prior capabilities through allocating additional task-specific mod- ules, leading to growing model capacity as new skills accumulate. Inspired by task arithmetic, we view the parameter change induced by downstream fine- tuning as a proxy for newly acquired capability. This recasts implicit skill re- tention as explicit parameter-space protection, avoiding both historical demon- stration replay and task-specific model expansion. Building on this view, we fur- ther identify that not every direction in a historical task vector deserves equal preservation: transferable task functionality can be entangled with environment- specific correlations, so indiscriminately protecting the full update may unnec- essarily restrict future adaptation. We therefore introduce CSD-VLA, a replay- free framework that uses task-preserving counterfactual interventions to identify an environment-sensitive subspace, decomposes each historical task vector into environment-sensitive and counterfactually invariant components, and selectively protects only the latter. Experiments on OpenVLA-OFT-7B and QwenOFT-3B across LIBERO and RoboTwin 2.0 demonstrate that CSD-VLA outperforms rep- resentative replay-based and replay-free baselines in overall continual-learning performance, strengthens retention under both task-level and cross-capability shifts, and better preserves historical skills under unseen visual changes, without replaying historical demonstrations or expanding the policy architecture.