- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies mid-execution instruction switching in vision-language-action (VLA) and world-action models. The authors show that policies often fail to sufficiently redirect their behavior after an instruction change, and describe this phenomenon as “trajectory hijacking.” They propose Counterfactual Action Steering (CAS), a training-free and model-agnostic inference-time method that compares target-conditioned actions with actions predicted under selected counterfactual instructions. CAS identifies potentially interfering tasks using the distance between the current end-effector position and task demonstration trajectories, and uses target–counterfactual action differences to modify the final action. Experiments on LIBERO-Goal-Switch, CALVIN-Custom-Switch, five VLA/WAM policies, and a real robot show consistent improvements. The paper also demonstrates that CAS-guided rollouts can provide useful self-training data.
Critical strengths
- This manuscript accurately positions the research problem of mid-execution instruction switching, particularly trajectory hijacking; the proposed CAS method is training-free and model-agnostic, concise and general, and has low deployment cost. 2.CAS yields significant improvements across five VLA/WAM policies, two instruction-switching benchmarks, and real-robot experiments, and the experiments are sufficient.
Critical weaknesses
1.“Trajectory hijacking” is not sufficiently established as the underlying mechanism.
The paper demonstrates that post-switch actions can remain correlated with source or non-target behaviors. However, the evidence does not establish that this is caused by an internal “trajectory prior.” The KNN distance between the current end-effector position and demonstration trajectories is only a geometric proxy; it does not show that the corresponding task representation is causally activated.
Alternative explanations include weak language–action coupling, visual dominance, insufficient action-expert grounding, and limitations of temporal state representation. These issues have been discussed in related VLA studies, including LIBERO-Plus [1], AVA-VLA [2], and VLA-Trace [3]. Therefore, trajectory hijacking is currently better supported as a behavioral description than as a validated mechanism.
2.The contribution is not clearly separated from general language-grounding interventions.
CAS improves switching performance, but it remains unclear whether it solves a specific trajectory-hijacking problem or generally compensates for weak language sensitivity. The paper does not sufficiently analyze the language encoder, cross-modal attention, or action expert.
The current ablations do not fully distinguish CAS from simpler alternatives such as target-action amplification, random counterfactual instructions, or matched action-norm corrections. Direct comparisons with language-grounding interventions or specialized switching methods would clarify the contribution.
Reference:
[1]Fei, Senyu, et al. "Libero-plus: In-depth robustness analysis of vision-language-action models." arXiv preprint arXiv:2510.13626 (2025).
[2]Xiao, Lei, et al. "AVA-VLA: improving vision-language-action models with active visual attention." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
[3]Shi, Haoyuan, et al. "VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing." arXiv preprint arXiv:2605.30117 (2026).
Questions for authors
1.While the behavioral improvements from CAS are empirically strong, the paper currently lacks mechanistic evidence to substantiate the 'trajectory hijacking' hypothesis. To strengthen this causal claim, could the authors provide internal analyses, such as cross-modal attention maps or representation probing across pre- and post-switch states, to explicitly demonstrate that CAS genuinely amplifies the target instruction's conditioning signal during action decoding?
2.The steering strength hyperparameter seems to vary significantly (e.g., 0.3 to 0.8 in real-world tests, 0.0013 to 0.035 in self-training). How sensitive is the method to this manual tuning, and does the linear extrapolation of actions risk severe kinematic overshooting or dynamics violations?
18 comments
Response to Question 1 (Part 2/4) (Empirical evidence for trajectory hijacking)
the paper currently lacks mechanistic evidence to substantiate the 'trajectory hijacking' hypothesis.
We would like to emphasize that our paper does provide substantial empirical evidence for trajectory hijacking. We also wish to clarify the distinction between this behavioral failure mode and its underlying explanation. As explicitly defined in Sec. 2.2, trajectory hijacking refers to post-switch behavior remaining fully or partially drawn toward non-target task trajectories, while visual trajectory priors are a key contributing factor we propose to explain this phenomenon. The paper supports this account with the following complementary observations and experiments.
1. Task-like behavior persists without task-specific language, but is substantially disrupted by visual intervention (Observation 2; Appendix D.1.1). We separately intervene on vision, language, and, where available, proprioception across five VLA and WAM policies on LIBERO. Visual inputs are replaced with a constant mean image, task instructions with "remain still", and proprioceptive inputs with the initial robot state. Disrupting vision causes the most severe performance collapse, even though object positions in LIBERO are nearly fixed. The resulting behavior often becomes erratic, repetitive, or noticeably reduced in magnitude. In contrast, removing task-specific language frequently preserves recognizable manipulation behavior: the robot may continue approaching and grasping an object, or execute a coherent competing task. These observations show that the current visual state can support structured task behavior even when the language no longer specifies that task.
2. Explicit stop supervision does not prevent continued motion during execution (Observation 2; Appendix D.1.2). To examine persistence beyond simply replacing the instruction with "remain still", we additionally finetune OpenVLA-OFT with 100 explicit remain-still trajectories, assigning zero commands to all six end-effector translation and rotation dimensions. We then test 50 online switches across ten source tasks and five switching steps, discarding any unexecuted source-conditioned actions at the switch.
Despite this supervision, the policy continues to move substantially. Over the first 20 post-switch steps, matching the length of the stop-training trajectories, the mean path length is 7.85 cm and the mean displacement from the switching state is 7.07 cm. After 100 steps, these reach 41.29 cm and 23.20 cm, respectively (Table 8). All 50 rollouts end more than 10 cm from the switching position. Thus, substantial motion persists even after the policy has been explicitly trained to associate the new instruction with stopping.
3. After an instruction switch, continued behavior can follow a non-target task trajectory (Observation 3; Appendix D.2). Fig. 4(a,b) illustrates a switch from grasping the bowl to turning on the stove. By the switch, the gripper is already close to the bowl. Although the new instruction requests stove operation, the robot continues all or part of the bowl-related behavior. The competing trajectory can belong to the source task or another non-target task.
Appendix D.2 quantifies the complete-execution form of this failure across policies and switching stages. In the CALVIN online evaluation, non-target task completion accounts for 17.5%, 17.0%, and 14.6% of target-task failures overall for OpenVLA-OFT, , and , respectively (Tables 11-12). For , this proportion rises to 31.4% in the latest switching range, . These statistics complement the examples of partial drift: the failure can involve continuing a coherent non-target task, not simply failing to complete the requested goal.
Together, these findings motivate the trajectory-prior explanation in Sec. 2.2. In standard demonstrations, states where the gripper is already close to an object are typically followed by continued approach or grasping, while counterfactual continuations are scarce. The policy can therefore learn to continue the familiar behavior suggested by the current gripper-object configuration, even after the instruction changes. We call this learned tendency a trajectory prior. The new internal analyses in Part 3 further examine how this persistence is reflected in attention and task-conditioned representations.
Response to Question 1 (Part 3/4) (Cross-modal attention and internal representations)
could the authors provide internal analyses, such as cross-modal attention maps or representation probing across pre- and post-switch states
Thank you for this thoughtful and valuable suggestion. We agree that examining policy internals can deepen our understanding of instruction-switching failures. Following your suggestion, we analyzed cross-modal attention maps to examine whether attention shifts toward the new target and how its responsiveness changes during execution. We also traced task-conditioned representations in to investigate where inside the policy different tasks become less distinguishable when the gripper is close to the source object. These analyses revealed findings beyond behavior alone and helped us understand how the persistence in Observations 2 and 3 is reflected in visual attention and task differences during action generation. We believe these insights have strengthened our paper. Our key findings follow.
1. Cross-modal attention analysis
Attention can remain on the previous target after an instruction switch. In OpenVLA-OFT, we change only the instruction, fixing the visual observation and robot state. We inspect final-layer attention from the positions used by the action head to image patches. In both examples in Fig. 3, attention remains prominent around the gripper and previous target even though the updated instruction requests a different object.
Attention becomes less responsive to instruction changes during execution. We compare attention changes at the initial state and after 24 source-task steps, using all 12 directed pairs of four tasks. Total variation (TV) measures how much the spatial attention distribution changes with the instruction within each camera view. After source-task execution, attention redistribution is smaller for every pair in both views. Mean TV decreases by 46.1% in the third-person view and 54.6% in the wrist view (Appendix D.1.3, Table 9). These comparisons use one shared initial configuration and one trajectory per source task. Thus, instruction changes affect attention less after the robot progresses toward the source task.
2. Layer-wise representation analysis
Action predictions become nearly indistinguishable when the gripper is close to the source object. Appendix D.3 investigates the action collapse described in Observation 3. Our main comparison follows the same wine-source trajectory at two stages. At , the gripper is still approaching the bottle and different instructions produce distinct action directions. At , the gripper is close to the bottle, yet instructions for the wine bottle, bowl, cream cheese, and stove produce nearly the same immediate action direction. Mean pairwise cosine similarity of the first predicted xyz action rises from 0.152 to 0.996. At each state, we change only the instruction, fixing the observation, robot state, and initial action noise.
Task differences persist throughout the language backbone. Even in the near-bottle state, the backbone produces distinct representations for the four tasks across all 18 layers (Fig. 15a). At its output, object-token separation remains 95% of its early-state value. The shared final newline token also develops instruction-dependent representations, retaining 90% of its early-state separation despite having the same input embedding under every instruction. The language backbone therefore still represents the task differences when the predicted actions have become almost indistinguishable.
Task differences become much weaker in the action expert. Following the same instructions into the action expert reveals a different pattern. The early state develops clear task separation during action generation, while separation is much weaker when the gripper is close to the bottle (Fig. 15b). This contrast is already present at the first flow update, when all instructions start from identical action noise. At the final update, expert-output separation in the near-bottle state is only 19% of its early-state value (Table 13). A second trajectory shows the same pattern: language object-token separation remains nearly unchanged, while final expert separation is 25% of its early-state value.
Together, these analyses give a more detailed account of the persistence described in the paper: attention can remain on the previous target after an instruction change, and, when the gripper is close to the source object, task differences preserved in the language backbone are expressed much less strongly in the action expert. The next part explains how CAS uses the remaining action differences to redirect execution.
Response to Question 1 (Part 1/4) (Overview of our response)
Thank you for this thoughtful question. We address its three aspects separately below.
Revisions. Observation 2, Fig. 3, and Appendix D.1.3 report the attention analysis; Observation 3 and Appendix D.3, "Where Do Task Distinctions Weaken during Action Generation?", report the representation analysis.