acceptodds
Under review as a conference paper at ICLR 2027

DeRail: Escaping Trajectory Priors via Counterfactual Action Steering for Mid-Execution Instruction Switching

36 views

Abstract

Robots operating in the real world must often adapt when user instructions change mid-execution, yet current instruction-conditioned robot policies, including Vision-Language-Action (VLA) models and World-Action Models (WAMs), are largely evaluated under fixed-instruction episodes. We study mid-execution instruction switching and identify trajectory hijacking as a key failure mode, where the policy fails to redirect sufficiently toward the switched target, with its post-switch behavior remaining fully or partially drawn toward trajectories associated with non-target tasks. We propose Counterfactual Action Steering (CAS), a training-free, model-agnostic inference-time method that identifies interfering non-target tasks, queries the same policy with corresponding counterfactual instructions, and uses target–counterfactual action differences to steer execution away from interfering trajectory priors. Experiments on LIBERO-Goal-Switch and CALVIN-Custom-Switch across five VLA and WAM policies, together with real-robot evaluation, show that CAS consistently improves switching success; for example, on CALVIN-Custom-Switch in the switching range, CAS improves from 5.8% to 54.6%. Successful CAS-guided rollouts further provide higher-quality self-training supervision than vanilla successful rollouts with the same number of trajectories, yielding stronger policies that generalize to unseen later instruction switches. Code will be made publicly available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, WAM, Instruction Switching, Counterfactual Action Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i94kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mid-execution instruction switching in vision-language-action (VLA) and world-action models. The authors show that policies often fail to sufficiently redirect their behavior after an instruction change, and describe this phenomenon as “trajectory hijacking.” They propose Counterfactual Action Steering (CAS), a training-free and model-agnostic inference-time method that compares target-conditioned actions with actions predicted under selected counterfactual instructions. CAS identifies potentially interfering tasks using the distance between the current end-effector position and task demonstration trajectories, and uses target–counterfactual action differences to modify the final action. Experiments on LIBERO-Goal-Switch, CALVIN-Custom-Switch, five VLA/WAM policies, and a real robot show consistent improvements. The paper also demonstrates that CAS-guided rollouts can provide useful self-training data.

Critical strengths
  1. This manuscript accurately positions the research problem of mid-execution instruction switching, particularly trajectory hijacking; the proposed CAS method is training-free and model-agnostic, concise and general, and has low deployment cost. 2.CAS yields significant improvements across five VLA/WAM policies, two instruction-switching benchmarks, and real-robot experiments, and the experiments are sufficient.
Critical weaknesses

1.“Trajectory hijacking” is not sufficiently established as the underlying mechanism.

The paper demonstrates that post-switch actions can remain correlated with source or non-target behaviors. However, the evidence does not establish that this is caused by an internal “trajectory prior.” The KNN distance between the current end-effector position and demonstration trajectories is only a geometric proxy; it does not show that the corresponding task representation is causally activated.

Alternative explanations include weak language–action coupling, visual dominance, insufficient action-expert grounding, and limitations of temporal state representation. These issues have been discussed in related VLA studies, including LIBERO-Plus [1], AVA-VLA [2], and VLA-Trace [3]. Therefore, trajectory hijacking is currently better supported as a behavioral description than as a validated mechanism.

2.The contribution is not clearly separated from general language-grounding interventions.

CAS improves switching performance, but it remains unclear whether it solves a specific trajectory-hijacking problem or generally compensates for weak language sensitivity. The paper does not sufficiently analyze the language encoder, cross-modal attention, or action expert.

The current ablations do not fully distinguish CAS from simpler alternatives such as target-action amplification, random counterfactual instructions, or matched action-norm corrections. Direct comparisons with language-grounding interventions or specialized switching methods would clarify the contribution.

Reference:

[1]Fei, Senyu, et al. "Libero-plus: In-depth robustness analysis of vision-language-action models." arXiv preprint arXiv:2510.13626 (2025).

[2]Xiao, Lei, et al. "AVA-VLA: improving vision-language-action models with active visual attention." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3]Shi, Haoyuan, et al. "VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing." arXiv preprint arXiv:2605.30117 (2026).

Questions for authors

1.While the behavioral improvements from CAS are empirically strong, the paper currently lacks mechanistic evidence to substantiate the 'trajectory hijacking' hypothesis. To strengthen this causal claim, could the authors provide internal analyses, such as cross-modal attention maps or representation probing across pre- and post-switch states, to explicitly demonstrate that CAS genuinely amplifies the target instruction's conditioning signal during action decoding?

2.The steering strength hyperparameter λ\lambda seems to vary significantly (e.g., 0.3 to 0.8 in real-world tests, 0.0013 to 0.035 in self-training). How sensitive is the method to this manual tuning, and does the linear extrapolation of actions risk severe kinematic overshooting or dynamics violations?

18 comments
AuthorsOpenReview ↗

Response to Question 1 (Part 1/4) (Overview of our response)

Thank you for this thoughtful question. We address its three aspects separately below.

  1. Existing empirical evidence (Part 2). Our paper has provide substantial empirical evidence for trajectory hijacking. We first summarize this evidence and clarify the distinction between the observed behavioral failure and visual trajectory priors as a contributing explanation.
  2. New internal analyses (Part 3). We took your suggestion seriously and conducted cross-modal attention analyses and task-conditioned representation tracing. These analyses have deepened our understanding of how attention responds to updated instructions and where task differences weaken during action generation.
  3. How CAS operates (Part 4). We clarify a potential misunderstanding: CAS applies a correction to the predicted action chunks. It amplifies target-counterfactual action differences at the output level, rather than modifying the target instruction's conditioning signal inside the action decoder.

Revisions. Observation 2, Fig. 3, and Appendix D.1.3 report the attention analysis; Observation 3 and Appendix D.3, "Where Do Task Distinctions Weaken during Action Generation?", report the representation analysis.

AuthorsOpenReview ↗

Response to Question 1 (Part 2/4) (Empirical evidence for trajectory hijacking)

the paper currently lacks mechanistic evidence to substantiate the 'trajectory hijacking' hypothesis.

We would like to emphasize that our paper does provide substantial empirical evidence for trajectory hijacking. We also wish to clarify the distinction between this behavioral failure mode and its underlying explanation. As explicitly defined in Sec. 2.2, trajectory hijacking refers to post-switch behavior remaining fully or partially drawn toward non-target task trajectories, while visual trajectory priors are a key contributing factor we propose to explain this phenomenon. The paper supports this account with the following complementary observations and experiments.

1. Task-like behavior persists without task-specific language, but is substantially disrupted by visual intervention (Observation 2; Appendix D.1.1). We separately intervene on vision, language, and, where available, proprioception across five VLA and WAM policies on LIBERO. Visual inputs are replaced with a constant mean image, task instructions with "remain still", and proprioceptive inputs with the initial robot state. Disrupting vision causes the most severe performance collapse, even though object positions in LIBERO are nearly fixed. The resulting behavior often becomes erratic, repetitive, or noticeably reduced in magnitude. In contrast, removing task-specific language frequently preserves recognizable manipulation behavior: the robot may continue approaching and grasping an object, or execute a coherent competing task. These observations show that the current visual state can support structured task behavior even when the language no longer specifies that task.

2. Explicit stop supervision does not prevent continued motion during execution (Observation 2; Appendix D.1.2). To examine persistence beyond simply replacing the instruction with "remain still", we additionally finetune OpenVLA-OFT with 100 explicit remain-still trajectories, assigning zero commands to all six end-effector translation and rotation dimensions. We then test 50 online switches across ten source tasks and five switching steps, discarding any unexecuted source-conditioned actions at the switch.

Despite this supervision, the policy continues to move substantially. Over the first 20 post-switch steps, matching the length of the stop-training trajectories, the mean path length is 7.85 cm and the mean displacement from the switching state is 7.07 cm. After 100 steps, these reach 41.29 cm and 23.20 cm, respectively (Table 8). All 50 rollouts end more than 10 cm from the switching position. Thus, substantial motion persists even after the policy has been explicitly trained to associate the new instruction with stopping.

3. After an instruction switch, continued behavior can follow a non-target task trajectory (Observation 3; Appendix D.2). Fig. 4(a,b) illustrates a switch from grasping the bowl to turning on the stove. By the switch, the gripper is already close to the bowl. Although the new instruction requests stove operation, the robot continues all or part of the bowl-related behavior. The competing trajectory can belong to the source task or another non-target task.

Appendix D.2 quantifies the complete-execution form of this failure across policies and switching stages. In the CALVIN online evaluation, non-target task completion accounts for 17.5%, 17.0%, and 14.6% of target-task failures overall for OpenVLA-OFT, π0\pi_0, and π0.5\pi_{0.5}, respectively (Tables 11-12). For π0\pi_0, this proportion rises to 31.4% in the latest switching range, [21,27][21,27]. These statistics complement the examples of partial drift: the failure can involve continuing a coherent non-target task, not simply failing to complete the requested goal.

Together, these findings motivate the trajectory-prior explanation in Sec. 2.2. In standard demonstrations, states where the gripper is already close to an object are typically followed by continued approach or grasping, while counterfactual continuations are scarce. The policy can therefore learn to continue the familiar behavior suggested by the current gripper-object configuration, even after the instruction changes. We call this learned tendency a trajectory prior. The new internal analyses in Part 3 further examine how this persistence is reflected in attention and task-conditioned representations.

AuthorsOpenReview ↗

Response to Question 1 (Part 3/4) (Cross-modal attention and internal representations)

could the authors provide internal analyses, such as cross-modal attention maps or representation probing across pre- and post-switch states

Thank you for this thoughtful and valuable suggestion. We agree that examining policy internals can deepen our understanding of instruction-switching failures. Following your suggestion, we analyzed cross-modal attention maps to examine whether attention shifts toward the new target and how its responsiveness changes during execution. We also traced task-conditioned representations in π0.5\pi_{0.5} to investigate where inside the policy different tasks become less distinguishable when the gripper is close to the source object. These analyses revealed findings beyond behavior alone and helped us understand how the persistence in Observations 2 and 3 is reflected in visual attention and task differences during action generation. We believe these insights have strengthened our paper. Our key findings follow.

1. Cross-modal attention analysis

Attention can remain on the previous target after an instruction switch. In OpenVLA-OFT, we change only the instruction, fixing the visual observation and robot state. We inspect final-layer attention from the positions used by the action head to image patches. In both examples in Fig. 3, attention remains prominent around the gripper and previous target even though the updated instruction requests a different object.

Attention becomes less responsive to instruction changes during execution. We compare attention changes at the initial state and after 24 source-task steps, using all 12 directed pairs of four tasks. Total variation (TV) measures how much the spatial attention distribution changes with the instruction within each camera view. After source-task execution, attention redistribution is smaller for every pair in both views. Mean TV decreases by 46.1% in the third-person view and 54.6% in the wrist view (Appendix D.1.3, Table 9). These comparisons use one shared initial configuration and one trajectory per source task. Thus, instruction changes affect attention less after the robot progresses toward the source task.

2. Layer-wise representation analysis

Action predictions become nearly indistinguishable when the gripper is close to the source object. Appendix D.3 investigates the action collapse described in Observation 3. Our main comparison follows the same wine-source trajectory at two stages. At t=10t=10, the gripper is still approaching the bottle and different instructions produce distinct action directions. At t=25t=25, the gripper is close to the bottle, yet instructions for the wine bottle, bowl, cream cheese, and stove produce nearly the same immediate action direction. Mean pairwise cosine similarity of the first predicted xyz action rises from 0.152 to 0.996. At each state, we change only the instruction, fixing the observation, robot state, and initial action noise.

Task differences persist throughout the language backbone. Even in the near-bottle state, the backbone produces distinct representations for the four tasks across all 18 layers (Fig. 15a). At its output, object-token separation remains 95% of its early-state value. The shared final newline token also develops instruction-dependent representations, retaining 90% of its early-state separation despite having the same input embedding under every instruction. The language backbone therefore still represents the task differences when the predicted actions have become almost indistinguishable.

Task differences become much weaker in the action expert. Following the same instructions into the action expert reveals a different pattern. The early state develops clear task separation during action generation, while separation is much weaker when the gripper is close to the bottle (Fig. 15b). This contrast is already present at the first flow update, when all instructions start from identical action noise. At the final update, expert-output separation in the near-bottle state is only 19% of its early-state value (Table 13). A second trajectory shows the same pattern: language object-token separation remains nearly unchanged, while final expert separation is 25% of its early-state value.

Together, these analyses give a more detailed account of the persistence described in the paper: attention can remain on the previous target after an instruction change, and, when the gripper is close to the source object, task differences preserved in the language backbone are expressed much less strongly in the action expert. The next part explains how CAS uses the remaining action differences to redirect execution.

Reviewer_pvegOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies mid-execution instruction switching and identifies trajectory hijacking as a recurring failure pattern. It proposes Counterfactual Action Steering (CAS), which contrasts target-conditioned and competing-task actions, using demonstration-trajectory proximity for counterfactual selection and weighting. Evaluation covers five policies, two simulation benchmarks, real-robot experiments, and self-training with successful CAS-guided trajectories.

Critical strengths
  1. CAS improves reported switching success across the evaluated VLA and WAM policies, with substantial gains in several CALVIN settings. This provides useful evidence that instruction-conditioned action contrasts can improve task switching.
  2. Randomized placements, selected-setting seed checks, fixed-weight sweeps, and matched self-training budgets strengthen the empirical assessment. Evaluating self-trained policies without CAS also demonstrates benefits incorporated into the policy itself.
Critical weaknesses
  1. The methodological contribution is not sufficiently isolated. The update closely resembles established contrastive guidance. In the specific setting of Table 5, Source-only CAS reaches 35.4%, versus 45.4% for full CAS, but task selection and weighting change together. Norm-matched action-scaling and random-counterfactual controls are not reported in the reviewed PDF. The modality interventions also do not isolate trajectory priors from general dependence on visual information.
  2. The evaluation supports a narrower conclusion than broad applicability. Real-robot testing covers one AIRBOT Play arm, one π₀ policy, and red/blue variants of block lifting, with 20 trials per method and setting. The main LIBERO evaluation uses one shared initial state, while the main CALVIN comparison starts from scripted intermediate states. Moreover, matching self-training trajectory counts does not fully isolate trajectory quality from task/state coverage, although Appendix F.4 provides partial supporting analysis.
Questions for authors
  1. What independent validation protocol was used to select CAS hyperparameters, particularly the switching-time-specific real-robot steering strengths?
  2. Does CAS outperform target-action scaling and random counterfactual instructions when correction magnitudes are matched?
  3. Does CAS-ST retain its advantage after controlling the retained trajectories’ source–target distribution, switching-state difficulty, and training-frame budget?
  4. Does the method generalize to more complex tasks on real robots? The real-robot experiments presented in the paper involve overly simple tasks.
15 comments
AuthorsOpenReview ↗

Response to Question 1 (Part 1/2) (Validation protocol and steering-strength sensitivity)

Thank you for this question. In simulation, all CAS hyperparameters were selected on held-out validation configurations disjoint from the evaluation configurations, and were then kept fixed for the final evaluation. On the real robot, where trials are expensive, λ\lambda was set more coarsely from a few pilot trials, as described below.

  • LIBERO-Goal-Switch: hyperparameters were tuned under the identical switching protocol from a different official initial state (the last official initial state associated with task 9), and evaluated from the shared initial state used in all main experiments.
  • CALVIN-Custom-Switch: hyperparameters were tuned on a separate set of intermediate switching states generated with a different random seed, which are disjoint from the evaluation states.
  • Real robot: Because each real-robot trial is costly, we did not run a systematic sweep. Instead, λ\lambda was coarsely adjusted using a small number of pilot trials, which were conducted separately from, and are not included in, the reported evaluation trials.

This protocol is now stated in Sec. 4.1 and detailed in the revised Appendices C.2, C.4 and H.2 (highlighted in blue).

Sensitivity to λ\lambda. The original submission already included a hyperparameter study (Sec. 4.4; Appendix E.6.2 in the revised paper). In that study, we varied λ\lambda, KK and δ\delta on the [15,21][15,21] switching range, and CAS outperformed the base policy in every tested setting. Following your question, we have now run a larger-scale λ\lambda sweep: λ\lambda from 0.02 up to 0.08 (more than 2.5$\times$ the largest value tested before) on all three switching ranges. The conclusion is unchanged and now holds much more broadly. CAS clearly improves over the base policy for every tested λ\lambda in every switching range, and a single λ\lambda works for all ranges. The extended sweep is discussed in Appendix E.6.2 (π0\pi_0 on CALVIN-Custom-Switch, K=30K=30, δ=0.1\delta=0.1; success rate in %):

λ\lambda[9,15][9,15][15,21][15,21][21,27][21,27]
0 (Base)22.16.70.0
0.0266.339.216.3
0.0362.545.417.5
0.0462.547.120.4
0.0562.545.820.0
0.0660.440.420.0
0.0761.740.019.6
0.0860.840.420.4

The table supports two observations:

  1. Large gains across all 21 settings. All seven values of λ\lambda improve over the base policy in all three switching ranges. Even the worst λ\lambda in each range (60.4 / 39.2 / 16.3) remains far above the base policy (22.1 / 6.7 / 0.0), with margins of +38.3, +32.5 and +16.3 percentage points. The gains are largest in the harder ranges:
    • On [15,21][15,21], every λ\lambda improves the success rate by a factor of roughly 6–7 (from 6.7% to 39.2–47.1%).
    • On [21,27][21,27], the base policy fails completely (0.0%), while every λ\lambda recovers 16.3–20.4% success.
  2. A single λ\lambda already works for all switching ranges. For example, λ=0.04\lambda=0.04 yields 62.5 / 47.1 / 20.4 versus 22.1 / 6.7 / 0.0 for the base policy, i.e., +40.4 / +40.4 / +20.4 points. Large gains therefore do not require tuning λ\lambda for each switching time; per-setting selection on held-out configurations mainly refines them. Moreover, for λ∈[0.03,0.05]\lambda \in [0.03, 0.05], performance varies by at most 2.9 points within each range (62.5 / 45.4–47.1 / 17.5–20.4), so CAS does not hinge on a precisely tuned value.

Tolerance to small positional errors. We additionally test whether small changes in gripper position affect subsequent task completion. On three LIBERO-Goal tasks, we apply 1–3 cm translations within the same task neighborhood and let OpenVLA-OFT continue under the original instruction without CAS. Completion remains at 92–100%, compared with 94–100% without perturbation (50 trials per task and perturbation magnitude; Appendix D.4). These results suggest that, once execution reaches a suitable interaction neighborhood, small positional errors need not prevent completion. This local tolerance may help explain why a range of steering strengths remains effective in the λ\lambda sweep above.

AuthorsOpenReview ↗

Response to Question 1 (Part 2/2) (Other hyperparameters and revisions)

Other hyperparameters. In simulation, K=30K=30 and δ=0.1\delta=0.1 are shared across all policies and benchmarks without per-setting tuning; on the real robot we use δ=0.03\delta=0.03 (Appendix H.2). Across the tested range, CAS consistently outperforms the base policy for both hyperparameters (Appendix E.6.2): performance is nearly constant across KK, and every tested δ\delta clearly outperforms the base policy, with the default not being the best-performing value on the evaluation set.

Revisions (all highlighted in blue in the revised PDF):

  • Sec. 4.1: stated the held-out tuning protocol.
  • Sec. 4.4 (Ablation Study): summarized the robustness across all three switching ranges and referred readers to Appendix E.6.2.
  • Appendices C.2 / C.4: added the hyperparameter selection protocol for CALVIN-Custom-Switch and LIBERO-Goal-Switch.
  • Appendix E.6.2: presented the extended λ\lambda sweep across all three switching ranges alongside the original hyperparameter study.
  • Appendix D.4: added local positional perturbation results showing that task completion remains high under small gripper-position changes.
  • Appendix H.2: added how λ\lambda was set on the real robot.
AuthorsOpenReview ↗

Response to Question 3 (Matched self-training comparison)

Thank you for asking whether CAS-ST retains its advantage after controlling task composition, switching-state difficulty, and training-frame budget. We conducted an additional paired self-training experiment with π0.5 on LIBERO-Goal, matching the switching states, source–target task pairs, and trajectory counts per task pair. CAS-ST retains its advantage while using fewer dataset frames.

For each of the 10 tasks, we execute the base policy for 25 actions from 10 distinct initial states, then switch each resulting prefix to the other nine target tasks, yielding 900 switching cases. Vanilla and CAS branch from the same saved simulator state, initial observations, and policy seed. We retain 223 jointly successful pairs, with initial target-action predictions aligned before CAS is applied. The two training sets therefore contain successful post-switch trajectories from exactly the same switching cases, with identical task-pair composition and counts.

Each set is combined with the same 500 official demonstrations. Both policies are fine-tuned from the same base policy for 10,000 steps, with batch size 32 and seed 42.

Training groupPost-switch trajectoriesOfficial demonstrationsPost-switch framesTotal dataset frames
Vanilla-ST22350027,49591,223
CAS-ST22350023,88787,615

Evaluated without test-time CAS on the same 90 off-diagonal task pairs at each switching time, CAS-ST outperforms Vanilla-ST at all four settings, including the later switches not used for data collection. Overall success improves from 34.2% to 41.9% (+7.8 percentage points).

Switching timeVanilla-ST (%)CAS-ST (%)Gain (percentage points)
t=2544.451.1+6.7
t=3031.144.4+13.3
t=3532.238.9+6.7
t=4028.933.3+4.4
Overall34.241.9+7.8

Overall pools 360 episodes per method; gains use unrounded rates.

Regarding the frame budget, CAS-ST uses 13.1% fewer post-switch frames and 4.0% fewer total dataset frames, with all selected trajectories retained in full, without truncation or downsampling. Thus, the comparison matches trajectory counts and optimization steps, while CAS-ST uses fewer unique dataset frames. The dataset frame counts themselves are not exactly matched.

Complementary distribution and recovery analyses. We also analyze the original datasets, each containing 148 trajectories. An energy two-sample test finds no statistically detectable overall difference in switching-state position, orientation, and gripper features (source-stratified p=0.612p=0.612; source-target-stratified p=0.910p=0.910). The paired experiment above controls the starting states directly. Trajectory-length and action analyses further show how the successful behaviors differ: in the paired dataset, CAS completes 173 of 223 cases (77.6%) in fewer steps and changes the translational command direction by an average of 10.13 degrees relative to the unsteered target prediction. Appendix H.3 provides the full state, length, and action statistics. Qualitative examples in Appendix F.1 show more suitable contact or grasp poses and fewer unnecessary recovery motions, consistent with the shorter successful trajectories.

The improvement therefore persists under matched switching states and task composition, with fewer dataset frames and the same fine-tuning budget. This supports the training value of CAS-guided trajectories under controlled starting conditions; subsequent state visitation remains shaped by the resulting recovery behavior.

Revision. Appendix F.7 presents the paired self-training protocol, dataset frame counts, and success-rate comparisons, with a summary and reference in Section 4.3. Appendix H.3 reports the complementary switching-state distribution, trajectory-length, and action analyses; Appendix F.1 provides the qualitative recovery comparisons.

Reviewer_dw9vOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates mid-execution instruction switching in vision-language-action (VLA) and world-action models (WAM), identifying Trajectory Hijacking as a key failure mode where non-target trajectory priors interfere with adapting to new instructions. To address this issue, the authors propose Counterfactual Action Steering (CAS), a training-free, model-agnostic method that identifies interfering tasks and uses differences between target-conditioned and counterfactual actions to correct the policy's behavior. The authors further introduce CAS-guided self-training (CAS-ST), which leverages successful corrected trajectories to improve policy adaptation. Experiments across five VLA/WAM policies on LIBERO-Goal-Switch and CALVIN-Custom-Switch, together with real-robot evaluations, demonstrate improved instruction-switching success rates. CAS-ST also outperforms conventional self-training with the same number of successful trajectories.

Critical strengths
  1. The method design is simple and has good model compatibility. CAS does not rely on a specific network architecture or internal model representation, nor does it require retraining the original policy. The authors construct an inference-time action correction signal by comparing the predicted actions for the target task and potential distractor tasks. The method is interpretable and simple to implement, and shows some generality across robot policies with different architectures.

  2. Experimental results are relatively comprehensive, with clear performance gains. The authors conduct experiments on two simulation benchmarks and five robot policies, and supplement them with real-robot evaluation. Across multiple instruction-switching stages, CAS improves the task success rates of the original policies, especially in the more difficult scenarios with later switches. In addition, the authors report inference latency and partial design ablations, which strengthen the credibility of the experimental results.

  3. The self-training extension has research value. Beyond immediate inference-time correction, the paper further explores how to improve the original policy using successful trajectories collected after correction. Experiments show that, with the same number of training trajectories, CAS-ST achieves better performance than conventional self-training and maintains some advantage under unseen later instruction switches. This provides valuable experimental evidence for using inference-time interventions to generate high-quality training data.

Critical weaknesses
  1. The source of CAS's effectiveness needs to be further distinguished from simple action-adjustment strategies. The main operation of CAS is to add a weighted difference between the target action and the counterfactual action to the target action. Mathematically, this is close to linear extrapolation in action space. Therefore, it is necessary to distinguish whether the performance gain of CAS comes from the useful difference information provided by the counterfactual task, or mainly from strengthening the target action direction or amplifying the action magnitude. The authors are suggested to add the following control experiments: (a) Amplifying only the magnitude of the target action without introducing a counterfactual action; (b) Constructing action differences using randomly selected non-target instructions; (c) Normalizing the direction of the action difference to rule out the effect of action-magnitude changes; (d) Comparing with other simple action interpolation or extrapolation strategies.

  2. Dependence on predefined task trajectories limits the applicability of the method. Full CAS requires pre-collected demonstration trajectories for candidate tasks and uses KNN distances between the current end-effector position and these trajectories to identify potential distractor tasks. However, spatial proximity of the end effector does not necessarily reflect the interference relationship between task behaviors. Different tasks may share part of their motion trajectories while having different manipulation goals; the same spatial position may also correspond to different task stages or object-interaction states. Moreover, real robots may face new tasks that are not predefined, in which case the candidate task trajectories required by full CAS may not be available. The authors are suggested to further analyze the method's performance under highly overlapping task trajectories, inaccurate distractor-task identification, and incomplete candidate tasks, and to clarify the dependency boundary of CAS on task priors and demonstration data.

  3. The source of CAS-ST's gains needs to be further disentangled from differences in training-data distribution. One of the important conclusions of this paper is that, compared with ordinary successful trajectories, successful trajectories generated under CAS guidance can provide higher-quality self-training supervision. The authors compare methods while controlling the number of successful training trajectories, which is a reasonable experimental design. However, the same number of successful trajectories does not imply that the training data have the same state distribution, action distribution, or task difficulty. Therefore, the performance gain of CAS-ST may come from more effective recovery behaviors, or partly from richer state coverage or a different training-data distribution. The authors are suggested to further analyze the differences between CAS-ST and Vanilla-ST training data in terms of state coverage, trajectory length, action distribution, and recovery behavior, and to verify through appropriate matching experiments whether the performance gain mainly comes from more informative correction behaviors. In addition, the current evidence for generalization mainly comes from unseen later switching times within the same task benchmark. It is suggested to distinguish generalization to unseen switching times from generalization to new tasks and new environments, and to avoid overly broad conclusions about transferability.

Questions for authors
  1. Compared with simple strategies such as merely strengthening the target action, using random counterfactual actions, and normalized action differences, does CAS still have a clear advantage?

  2. When multiple candidate tasks share similar trajectories, or when KNN incorrectly identifies distractor tasks, how much does CAS's performance degrade? What is the dependency boundary of the method on candidate task trajectories?

  3. Does the improvement of CAS-ST over Vanilla-ST mainly come from the correction behavior itself, or from differences in training data regarding state coverage and action distribution? Can the authors provide a data-distribution matching analysis?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1 (Action amplification and alternative action steering)

Thank you for suggesting these controls. We conducted supplementary experiments to examine whether the benefits of CAS can be explained by target-action amplification, arbitrary counterfactual instructions, or simple action interpolation. We also tested final-action norm matching and reversal of the CAS correction to examine the roles of action magnitude and correction direction.

MethodLIBERO, ts=20t_s=20CALVIN, [15, 21]
Base policy30.06.7
Full CAS37.845.4
Target-action scaling (α=1.5\alpha=1.5)20.02.9
Target-action scaling (α=2\alpha=2)21.11.3
Target-action scaling (α=3\alpha=3)18.90.4
Random counterfactuals (KNN-based weighting)25.622.9
Random counterfactuals (fixed weighting, w=0.6w=0.6)22.222.1
CAS with reversed correction direction14.40.0
CAS with final-action norm matching40.042.5
Random counterfactual action interpolation (β=0.5\beta=0.5)8.92.9

Target-task success rates (%) for π0\pi_0 on LIBERO-Goal-Switch (ts=20t_s=20) and CALVIN-Custom-Switch (ts∈[15,21]t_s\in[15,21]). Higher is better. The complete comparison is reported in Table 21 of Appendix E.6.3.

Target-action amplification and random counterfactuals. Scaling the target action by α∈{1.5,2,3}\alpha\in\{1.5,2,3\} yields 18.9–21.1% success on LIBERO and 0.4–2.9% on CALVIN. Replacing the selected counterfactual with a randomly sampled non-target instruction yields 25.6%/22.9% with KNN-based weighting and 22.2%/22.1% with fixed weighting. These alternatives do not reproduce the performance observed with CAS, supporting the value of informative counterfactual actions beyond simply strengthening the target action.

For both random-counterfactual variants, we uniformly sample one instruction from all non-target tasks at each replanning step without applying the distance-threshold gate δ\delta, so that distance-based filtering does not restrict random selection. KNN distances determine only the steering weight in the adaptive variant. On CALVIN, the improvement over Base may partly reflect the small task set: excluding the target leaves only three candidates, making it relatively likely to sample a task that provides a useful correction. On LIBERO, however, random counterfactuals with KNN-based weighting reduce success from 30.0% to 25.6%, suggesting that random selection can introduce interference.

Action magnitude and direction. Because the environment executes the final action, we define this magnitude control at the output level. We rescale the final CAS action to the norm of the target-conditioned action at the same observation, preserving the CAS output direction. This achieves 40.0%/42.5% without increasing the final six-dimensional action norm. The norm-matched variant outperforms full CAS on LIBERO but underperforms it on CALVIN; CAS therefore remains effective after final-action amplification is removed. Reversing the CAS correction while preserving its magnitude yields only 14.4%/0.0%, further supporting the importance of the correction direction. This directly controls the magnitude of the action applied to the environment.

Simple interpolation. Equal-weight interpolation between target-conditioned and random non-target actions achieves only 8.9%/2.9%. Together, these results support a role for the content and direction of the counterfactual correction, rather than larger final actions or arbitrary action mixing alone.

Revisions. We added the complete comparison in Appendix E.6.3, Comparison of Alternative Action Steering Strategies (Table 21), and summarized the findings in Sec. 4.4 with a pointer to the appendix. The appendix defines each alternative, clarifies that the random-counterfactual variants do not use the δ\delta gate, and discusses their different effects on LIBERO and CALVIN. All additions are highlighted in blue.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 1/2) (Trajectory overlap and candidate-task dependencies)

Thank you for raising this point. We distinguish the information required by full CAS from its lightweight Source-only variant, and examine how shared motions and incomplete candidate coverage affect steering.

Dependency on candidate task trajectories. Full CAS does not require demonstration trajectories for all tasks encountered during pretraining. Instead, it relies on a predefined library of candidate tasks and their demonstration trajectories. In our experiments, these come from downstream task data, including demonstrations for LIBERO-Goal and CALVIN-Custom-Switch, as well as the training demonstrations for the red- and blue-block lifting tasks in our real-robot experiments.

Applicability without a trajectory library. Our lightweight Source-only CAS instead uses only the known source and target instructions, requiring neither demonstration trajectories nor an enumerated library of other tasks. In Table 5, it raises CALVIN Switch SR from 6.7% to 35.4%, compared with 45.4% for full CAS. It therefore provides a practical alternative when a candidate library is unavailable, while still relying on the underlying policy's ability to produce meaningful instruction-conditioned actions. Full CAS offers the additional benefit of correcting interference from tasks beyond the source when their demonstrations are available.

Shared trajectories and correction direction. Candidate task ii contributes wi(atgt−ai)w_i(a_{\mathrm{tgt}}-a_i), with both actions predicted from the same current observation. End-effector proximity determines the weight, while the action difference determines the correction direction. Identical predictions during a shared stage yield zero correction; similar predictions yield a small correction. Thus, when different tasks share a trajectory and predict the same action, their shared motion does not introduce an incorrect correction direction.

LIBERO-Goal-Switch contains natural examples of shared manipulation stages and spatial overlap: bowl tasks share approaching and grasping the same bowl before placing it on the stove, cabinet top, or plate; wine-bottle tasks share grasping before moving to the cabinet top or rack; and the bowl-to-cabinet and bottle-to-cabinet tasks share a placement region despite manipulating different objects. We analyze switching success when these tasks are the targets:

PolicyBowl targetsWine-bottle targetsCabinet-top targets
OpenVLA-OFT38.0 → 44.44.2 → 13.916.7 → 23.6
π0\pi_048.1 → 49.127.8 → 44.443.1 → 50.0
π0.5\pi_{0.5}81.5 → 86.140.3 → 58.359.7 → 69.4
Fast-WAM76.9 → 73.130.6 → 45.855.6 → 68.1
LingBot-VA89.8 → 92.641.7 → 54.266.7 → 75.0

Entries show Base → CAS success rates (%), aggregated over ts∈{10,15,20,25}t_s\in\{10,15,20,25\}. Each target includes all nine other source tasks: 108 episodes per method for bowl targets and 72 for each other group. The groups share some targets and are reported separately. CAS improves 14 of the 15 policy-group combinations, with the exception of Fast-WAM on bowl targets. These results show that CAS remains useful for targets with shared manipulation stages or placement regions.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 2/2) (Inaccurate distractor-task identification and incomplete candidate tasks)

Inaccurate distractor-task identification. Providing accurate ground-truth labels for distractor tasks is challenging because precisely defining which tasks interfere at a given state is itself difficult. This makes distractor-task misidentification difficult to quantify directly. We therefore examine the effect of distractor-task choice through selection ablations. Source-only CAS always uses the known source task, while All non-target Fixed includes every non-target task without KNN-based filtering. Table 5 compares these alternatives with KNN-selected Fixed and full CAS:

MethodCounterfactual task setTask trajectory dataSteering weightSwitch SR (%)
Base policyNoneNoneNone6.7
Source-only CASSource taskNonew=0.6w=0.635.4
All non-target FixedAll non-target tasksNonew=0.4w=0.423.3
KNN-selected FixedKNN-selected non-target tasksAll candidate tasksw=0.6w=0.643.33
Full CASKNN-selected non-target tasksAll candidate tasksAdaptive45.4

Results use π0\pi_0 on CALVIN-Custom-Switch at ts∈[15,21]t_s\in[15,21]. For all three fixed-weight variants, we sweep w∈{0.1,0.2,0.4,0.6,0.8}w\in\{0.1,0.2,0.4,0.6,0.8\} and report the best result (Appendix E.6.1). Both Source-only CAS and All non-target Fixed improve over Base, although they underperform full CAS. Thus, useful correction remains possible with these alternative candidate sets. KNN-selected Fixed further improves success to 43.3%, supporting the value of selecting state-relevant competing tasks; distance-adaptive weighting raises it to 45.4%.

Sensitivity to candidate selection. We assess sensitivity to candidate choice through a controlled stress test: we replace trajectory-based selection with a uniformly sampled non-target instruction at each replanning step, without distance-threshold filtering, while retaining KNN-based weighting. This achieves 25.6% on LIBERO and 22.9% on CALVIN, compared with 37.8% and 45.4% for full CAS, using π0\pi_0 at ts=20t_s=20 and ts∈[15,21]t_s\in[15,21], respectively. These results show that arbitrary candidate selection is less effective than the selection used by CAS, supporting the importance of state-relevant counterfactuals.

Incomplete candidate coverage. To directly examine incomplete candidate coverage, we restrict the candidate library to a fixed half of the tasks: five of ten on LIBERO and two of four on CALVIN. The current target is excluded from the selected pool. We keep the full evaluation set and the original KNN threshold and weighting rule; only the available counterfactual tasks change. The results use π0\pi_0:

MethodLIBERO, ts=20t_s=20CALVIN, [15,21][15,21]
Base policy30.06.7
Full CAS37.845.4
CAS with half candidates33.3329.58

Success rates are reported in percent, with 90 episodes on LIBERO and 240 on CALVIN. With this half-sized candidate library, CAS still improves over Base on both benchmarks, reaching 33.33% and 29.58%, respectively. Full CAS performs better in both settings, indicating that broader coverage of competing tasks provides additional benefit. These results show that CAS can remain useful with an incomplete candidate library.

Revision. Table 5 and Appendix E.6.1 include the new KNN-selected Fixed comparison, separating task selection from adaptive weighting. We have clarified in Section 3.2 that the demonstration library covers a predefined set of downstream candidate tasks (e.g., LIBERO-Goal) and need not include all tasks encountered during pretraining. Together with the Source-only variant described in the same section, this makes the dependency on task priors and demonstration data explicit. The shared-trajectory target-group analysis and half-candidate-library results above are supplementary results reported in this response; they are not included in the current revised manuscript.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chairs (ACs),

We sincerely appreciate your time and effort in reviewing our paper.

During the rebuttal and discussion period, we addressed the concerns raised by all three reviewers with substantial new experiments, deeper behavioral and internal analyses, controlled methodological ablations, and expanded evaluations. These additions significantly strengthen the revised paper, with newly incorporated material highlighted in blue.

Reviewer pveg explicitly acknowledged that the rebuttal “has meaningfully strengthened the paper and resolved several concerns underlying my original review.” We also provided detailed responses and follow-ups to Reviewers dw9v and i94k.

ReviewerInitial RatingConfidencePost-rebuttal Feedback
dw9v32No further feedback
pveg23Acknowledged substantial strengthening and resolution of several concerns
i94k23No further feedback

Our revisions reinforce three connected contributions: (1) identifying and understanding trajectory hijacking, (2) correcting it through state-relevant counterfactual action steering, and (3) using CAS-guided rollouts to provide stronger self-training supervision. The following summaries highlight the reviewers' questions, the evidence we added, and the corresponding manuscript revisions.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Rebuttal (Part 1/5): Reviewer dw9v (Initial Rating: 3, Confidence: 2)

Reviewer dw9v highlights the simplicity, interpretability, and model compatibility of CAS, its broad experimental evaluation over five VLA/WAM policies and real-robot trials, and the research value of CAS-guided self-training.

Weakness 1 / Question 1: Are CAS gains due to informative counterfactuals or simply larger actions?

  • Solution: We added target-action scaling, randomly selected counterfactuals, matched final-action norms, reversed correction directions, and random action interpolation (Appendix E.6.3, Table 21). Scaling the target action gives only 18.9–21.1% on LIBERO and 0.4–2.9% on CALVIN, versus 37.8%/45.4% for full CAS. Critically, norm-matched CAS achieves 40.0%/42.5% without increasing the final action norm; reversing its correction yields just 14.4%/0.0%.
  • Conclusion: The benefit depends on the information and direction in state-relevant counterfactual action differences, not merely action amplification or arbitrary mixing.

Weakness 2 / Question 2: Trajectory-library dependence, overlapping tasks, and incomplete candidate coverage.

  • Solution: We clarified that full CAS only needs a predefined library of downstream candidate tasks, not all tasks seen during pretraining (Sec. 3.2). Source-only CAS needs no demonstration trajectory library and already improves CALVIN success from 6.7% to 35.4%. Further analyses show that: (i) CAS improves 14/15 policy–target-group combinations involving shared grasping stages or overlapping placement regions; (ii) with half the candidate tasks, it still improves success from 30.0% to 33.3% on LIBERO and 6.7% to 29.6% on CALVIN; and (iii) KNN-selected Fixed achieves 43.3%, versus 23.3% for All non-target Fixed and 45.4% for full CAS (Table 5; Appendix E.6.1). The overlap and half-library studies were reported additionally in the rebuttal.
  • Conclusion: CAS remains useful with trajectory overlap or incomplete libraries, while the ablations identify state-relevant task selection as a major source of improvement and adaptive weighting as a complementary refinement.

Weakness 3 / Question 3: Does CAS-ST improve because of recovery quality or differences in training-data coverage?

  • Solution: We conducted a strictly paired self-training study with 223 jointly successful trajectory pairs, matching switching states, source–target assignments, and trajectory counts per task pair. With identical fine-tuning settings and no test-time CAS, CAS-ST improves overall success from 34.2% to 41.9% (+7.8 points) across four switching times, while using 13.1% fewer post-switch frames (Appendix F.7). Additional distributional analyses find no statistically detectable overall difference in the original datasets' switching-state proprioceptive distributions (Appendix H.3). CAS trajectories are shorter in 173/223 matched cases (77.6%), and qualitative comparisons show more suitable contact/grasp configurations and fewer unnecessary recovery motions (Appendix F.1).
  • Conclusion: Under controlled starting states and task composition, CAS-guided successes provide more effective de-hijacking supervision than vanilla successes, even with fewer frames. We also clarified that the measured generalization concerns unseen later switching times, not unseen tasks or environments.

Overall, the rebuttal directly addresses all three topics with new action-space controls, candidate-selection and overlap analyses, and a tightly controlled self-training comparison.

AuthorsOpenReview ↗

Summary of Rebuttal (Part 2/5): Reviewer pveg (Initial Rating: 2, Confidence: 3)

Reviewer pveg recognizes consistent CAS improvements across VLA/WAM policies, substantial gains on challenging CALVIN switches, and the value of our randomized-placement tests, robustness checks, and self-training comparisons evaluated without test-time CAS.

Weakness 1 / Question 2: Isolating CAS's methodological contribution from generic contrastive guidance.

  • Solution: We introduced KNN-selected Fixed to separate state-dependent counterfactual selection from adaptive weighting (Table 5; Appendix E.6.1). It achieves 43.3% success, compared with 23.3% for indiscriminate All non-target Fixed and 35.4% for Source-only CAS; full CAS reaches 45.4%. Extensive additional controls demonstrate that target-action scaling, random counterfactuals, and interpolation cannot reproduce full CAS, while norm-matched CAS retains 40.0%/42.5% on LIBERO/CALVIN (Appendix E.6.3). We further clarified the problem-driven insight: the central design decision is which interfering task behaviors should be contrasted at the current state, not the algebraic contrastive formula alone.
  • Conclusion: State-relevant counterfactual selection is a substantive component of CAS; adaptive weighting offers an inexpensive additional refinement. Our contribution also includes the trajectory-hijacking analysis and CAS-guided self-training, beyond inference-time guidance alone.

Question 1: Independent hyperparameter selection and sensitivity.

  • Solution: We documented held-out validation configurations disjoint from evaluation for both simulation benchmarks, and separate real-robot pilot trials for coarsely selecting steering strengths (Sec. 4.1; Appendices C.2, C.4, H.2). We added an extended sweep of seven nonzero steering strengths over three switching ranges (21 settings): every setting improves over the base policy, and a single value already works well across all three ranges (Appendix E.6.2). We additionally tested 1–3 cm gripper perturbations, retaining 92–100% original-task completion (Appendix D.4).
  • Conclusion: Strong CAS improvements do not depend on narrowly tuned steering strengths; the local positional tolerance offers additional insight into this robustness.

Weakness 2 / Question 4: Evaluation breadth and motivation for controlled real-world experiments.

  • Solution: We expanded LIBERO evaluation to different initial states for all 90 source–target pairs: CAS's +10.3-point gain closely matches +10.6 points under the main shared-state protocol (Appendix E.5). We also added fully online CALVIN switching from states actually reached through source-policy execution, obtaining gains of +50.0, +20.2, and +4.8 points across three switching ranges (Appendix E.7). Our physical evaluation deliberately uses matched red/blue block-lifting conditions to isolate responsiveness to a changed instruction from basic skill failures. Across 80 trials per method, CAS improves aggregate success from 27.5% to 65.0%, and verifies the method with joint-space actions rather than the end-effector actions used in simulation. We explain this controlled evaluation rationale in Appendix J.
  • Conclusion: The new evaluations establish robustness beyond shared starts and scripted switching states, while the physical experiments give measurable, controlled evidence for the steering principle.

Weakness 2 / Question 3: Self-training under matched task and state distributions.

  • Solution: Our new 223-pair controlled self-training experiment matches switching states, source–target pairs, and trajectory counts, using identical optimization settings (Appendix F.7). CAS-ST achieves 41.9% versus 34.2% for Vanilla-ST, despite using fewer frames. Complementary switching-state, action, and trajectory analyses appear in Appendix H.3.
  • Conclusion: CAS-ST's advantage persists after controlling the starting conditions and task composition, strengthening the evidence for the quality of CAS-guided recovery supervision.

Post-rebuttal feedback: Reviewer pveg explicitly stated that the rebuttal “has meaningfully strengthened the paper and resolved several concerns underlying my original review.” We then provided additional clarification of our problem-driven selection mechanism, behavioral insights, self-training contribution, and controlled physical evaluation design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.