acceptodds
Withdrawn from ICLR 2027

When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models can exceed 90% success on in-distribution tasks and withstand changes that preserve the required action, yet fail under changes that demand a different action. Aggregate robustness scores can therefore conceal a more specific failure, in which a policy responds to both language and vision yet does not combine them to select the action the task requires. We call this failure . Instructions cue familiar trajectory families, and visual feedback adjusts their execution. Behavioral analyses of fine-tuned and GR00T-N1.7 policies reveal that failed rollouts often retain the source behavior or switch to another demonstrated task. These switches show that language is not simply ignored. Readouts and interventions connect these choices to task-conditioned internal states. Our analysis of the imitation objective shows how narrow conditional action support can leave grounded and instruction-keyed solutions indistinguishable on the demonstrations. This motivates (ECT), which acts at two levels. supply valid demonstrations in which the same instruction requires different actions in distinguishable scenes, while the trains each demonstration with its counterpart in the same update. In a controlled LIBERO-PRO comparison, full ECT raises 's mean position-swap success from 36% to 59%. On CALVIN, where counterparts already occur in the original data, the ECT loss improves five-task completion without new demonstrations. On a real UR5e under a fixed demonstration budget, full ECT raises unseen-position success from 8% to 88%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EP2LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies instruction–action binding in vision-language-action models: instructions can trigger familiar task behaviors, while visual feedback adjusts execution, without correctly selecting the action required by the current instruction and scene.

Its contributions include behavioral and internal-state analyses of this failure in π0.5\pi_{0.5} and GR00T-N1.7, an explanation based on insufficient scene-dependent action diversity in demonstrations, and Equivariant Counterfactual Training (ECT). ECT supplies valid demonstrations where the same instruction requires different actions across scenes and trains corresponding examples together using standard imitation losses.

Experiments on LIBERO-PRO, CALVIN, and a real robot show improved generalization, particularly under position changes. Most gains come from counterfactual data, while paired training provides additional but inconsistent benefits.

Critical strengths
  • The paper provides a substantive diagnosis beyond aggregate success rates. Behavioral probes and internal-state analyses show that language sensitivity and visual tracking can coexist with incorrect task selection. This distinction is consequential because it challenges the use of modality responsiveness as evidence of grounded instruction following and identifies a concrete failure mode for evaluation and training.

  • The supervision principle receives complementary empirical support. Controlled LIBERO-PRO experiments separate the contributions of counterfactual data and paired training, while CALVIN demonstrates benefits from pairing existing demonstrations and the real-robot study shows gains under a fixed demonstration budget. Together, these results support the practical importance of scene-dependent action alternatives beyond simply collecting more demonstrations.

Critical weaknesses
  • The incremental contribution over closely related work is insufficiently established. Prior work already documents familiar-task execution under counterfactual instructions and proposes counterfactual supervision or geometric demonstration augmentation. Although the paper offers a more nuanced diagnosis, its experiments do not clearly distinguish instruction–action binding from existing shortcut-learning explanations or directly compare ECT with closely related methods. This makes it difficult to assess whether the proposed framework yields substantially new explanatory or methodological value.
  • The benefit of paired training is not sufficiently isolated. Most LIBERO-PRO gains come from counterfactual data, while pairing provides smaller, inconsistent improvements. The paired setup jointly changes counterpart co-presentation, original/counterfactual sampling balance, and shared flow-matching noise and time. Consequently, the experiments do not establish which ingredient drives the additional gains. Since paired training is a central methodological contribution, this attribution gap materially weakens the evidence for its novelty and effectiveness.
Questions for authors
  • How does ECT compare directly with existing demonstration-augmentation methods, such as RoCoDA, under matched data and training budgets? Without this comparison, how can we distinguish the benefits of ECT’s specific design from those of broader scene and action coverage?
  • Why does the third block of Table 2 report only ECT data results for GR00T-N1.7, without ECT loss? Was paired training evaluated but found ineffective, or was it omitted for other reasons? This distinction matters for assessing whether the full method generalizes across architectures.
3 comments

We thank the reviewer for the specific questions, which we answer with new controlled experiments. In short, benchmark scores hide a failure in which language selects a familiar trajectory and vision only tracks it. We trace it to supervision, and ECT supplies the missing supervision (see also the General Response).

  • W1, distinguishing binding from shortcut explanations. Binding is a specific shortcut that ignored-language and vision-prior accounts do not explain. Both modalities are active. With the target swapped and shifted by 10 cm, the arm follows the object in the source-bound slot, not the named one (Sec. 3), and in LIBERO-Goal 37% of rollouts switch to another demonstrated task. Training permits this, because an instruction-keyed policy fits demos in which each instruction covers a narrow action family (Sec. 2.1). Unlike CAG/CAST (instruction varied, scene fixed) and RoCoDA (no language), ECT fixes the instruction and changes the required action (Sec. 6).

    The choice is also visible inside the model. Reading π0.5's internal state identifies the wrongly executed task beyond word overlap, and injecting another task's state redirects the arm in both models (Sec. 4):

DiagnosticResultControl
Executed task ranked first, scene-matched vs text-only readout34/4522/45
π0.5, another task's state injected, arm redirected109/2000/200
GR00T, another task's state injected, arm redirected121/2000/200
  • W2, isolating co-presentation, sampling balance and shared noise/time. With the data fixed, balance explains most of the additional ECT loss gain on both benchmarks, and exact pairing adds a smaller gain on top. This sharpens our account. ECT data supply missing same-instruction action alternatives, and the ECT loss balances existing ones. Both implement the supervision principle derived from our diagnosis.

    We thank the reviewer for this question, which makes the paper clearer, and will revise Sec. 5.1 and Fig. 3 to present the ECT loss as balanced counterpart sampling. Matched updates, steps and data pool:

Same data, standard sampling+ balance (partners shuffled)+ co-presentationFull ECT loss (+ shared noise/time)
LIBERO, π0.5 Swap (%)56.158.757.758.9
CALVIN, avg. length3.604.23—4.35
  • Q1, RoCoDA under matched data and training budget. Swap means at 30K steps are 32.5 for action-invariant randomization, 48.6 for RoCoDA-style random-pose placement, and 56.1 for ECT data via mirroring.

  • Q2. Due to compute limits, paired GR00T training was not evaluated at submission. We now report it (3 seeds).

    As on π0.5 (Table 3), the LIBERO gain comes mainly from ECT data. The ECT loss shows its value where counterparts already exist, raising CALVIN five-task completion from 58% to 76% with no new demos (Table 4). GR00T results:

GR00T-N1.7SwapTask
ECT data19.724.5
+ ECT loss19.223.5

We hope this answers the reviewer's two central questions, namely what binding adds beyond generic shortcut accounts (which shortcut, how internal states drive it, why imitation permits it) and what each ECT-loss ingredient contributes. We would be grateful if the reviewer reconsidered their assessment of our contributions given these clarifications and results. We welcome further engagement.

I thank the authors for the additional experiments. They clarify that most gains come from counterfactual data and balanced sampling, while the independent benefit of pairing remains unestablished. My main concern about the methodological contribution therefore remains, and I retain my original rating.

We thank the reviewer for the initial question of whether the gain comes from pairing. Our submission left the mechanism of the ECT loss as an open question, and the reviewer’s question turned it into a concrete result, which we appreciate. In controlled experiments, balanced sampling is the main source of the additional gain. ECT already balances originals and counterparts through matched-pair sampling, so this clarifies an existing component, not a new method. We will revise Sec. 5.1 and Fig. 3 accordingly and no longer claim a pairing-specific contribution. The controls resolve the attribution concern. Our central contribution remains the diagnosis of instruction-action binding and the supervision principle it reveals. We respectfully ask the reviewer to reconsider their assessment and to specify which concerns remain unresolved.

Reviewer_CvLkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the generalization failure problem of Vision-Language-Action, proposing instruction-action binding as the core failure mode and introducing Equivariant Counterfactual Training to mitigate this problem. Effective improvements are achieved on multiple benchmark datasets.

Critical strengths
  1. The paper is logically structured, progressing step-by-step from problem diagnosis and mechanism analysis to method design and multi-scenario verification, making it highly readable.

  2. It addresses a core real-world pain point of VLA models—instruction-action binding—revealing that while the model appears to utilize both language and vision, it actually only recalls memory trajectories in a failure mode.

  3. The experimental evidence is comprehensive. Validation was conducted on simulation benchmarks, public datasets, and real robot platforms.

Critical weaknesses
  1. The proposed problem (the instruction retrieves a familiar trajectory seen during training) and its root cause (the same instruction often corresponds to only a narrow set of trajectories, resulting in insufficient support for conditional actions) have been reported in related research (Dismantling the Illusion of Vision-Language-Action Models Competence via Explicit Distributional Shifts).

  2. The ECT data construction in the proposed ECT method is obvious. The reason for this problem is the fixed binding of data. Referring to the libero-goal task, different task models can perform well under the same scenario.

  3. Regarding the ECT loss, the experiments did not fully demonstrate the advantages of the ECT loss; the improvement is still due to data augmentation.

Questions for authors

See above Weaknesses.

3 comments

We thank the reviewer for recognizing our diagnosis and multi-setting validation. In short, benchmark scores hide a failure in which language selects a familiar trajectory and vision only tracks it. We trace it to supervision, and ECT supplies the missing supervision (see also the General Response).

  • W1, "already reported" (LIBERO-Gen). LIBERO-Gen studies task × context coverage and shows qualitatively that policies keep the original trajectory. But coverage can grow through appearance alone, without changing any required action, and the failure is not visual blindness, since the arm follows a target shifted by 10 cm yet keeps the source-bound choice (Sec. 3).

    Beyond LIBERO-Gen, 15% of rollouts switch to another demonstrated task, internal states identify the executed task (34/45 vs 22/45 text-only) and drive it (injection redirects the arm in 109/200 vs 0/200, Sec. 4), and Sec. 2.1 derives why imitation permits this. The account thus predicts which shifts fail, why more of the same data does not help, and which supervision fixes it. We will cite and discuss LIBERO-Gen.

  • W2, "obvious". The fix looks obvious only after the diagnosis, and that is the point. Before it, common remedies failed. Shuffling spatial phrases did not improve Swap, and instruction dropout made it worse. Once we identified what VLA training misses, ECT followed naturally. This is the value of an analysis-driven paper. A precise diagnosis makes the remedy simple, and the review credits this diagnosis as a strength.

    Controls confirm this. 4× more data stays ≥34 points below ID (Table 25), and same-size instruction or appearance randomization gives nothing. Even on the reviewer's LIBERO-Goal example, where tasks share one scene and models succeed in-distribution, Swap stays at 41.3 and only ECT data moves it:

π0.5, LIBERO-GoalSwapTask
Standard41.320.6
+ instruction augmentation (same size)41.222.2
+ appearance randomization (same size)41.020.4
ECT data51.840.7
  • W3, "due to data augmentation". CALVIN isolates the ECT loss. With the data unchanged, no augmentation and no new demos, the loss alone raises five-task completion from 58% to 76% (Table 4). This gain cannot come from augmentation.

    The two components address one gap at two levels. ECT data construct same-instruction alternatives that require different actions when they are missing, which same-size appearance randomization does not (table above). The ECT loss makes the policy use such alternatives in balance when they exist, adding a further 2.8 Swap points on LIBERO with the data held fixed (Table 3). Our contribution is the diagnosis and the supervision principle it reveals. Both components implement that principle, so the value of ECT rests on the supervision it supplies, not on either component alone.

We hope this clarifies what our diagnosis adds beyond LIBERO-Gen and why a simple remedy follows. We would be grateful if the reviewer reconsidered their assessment of our contributions given these clarifications and results. We would value the reviewer's engagement.

Thank you for the detailed response and additional experiments.

However, I remain unconvinced that the response sufficiently addresses my primary concerns regarding the novelty of the proposed method. While the additional experiments provide useful insights, they do not fundamentally change my assessment that the core ECT data construction follows relatively straightforwardly from the identified failure mode, and that the overall methodological contribution remains limited.

I acknowledge the additional effort made by the authors, but my main concerns remain. Therefore, I will maintain my original rating.

We thank the reviewer for engaging. That ECT follows naturally from the diagnosis does not make the diagnosis obvious or the contribution limited. Our diagnosis explains why narrow action support under each instruction allows an instruction-keyed policy to fit demonstrations, and predicts which supervision can disrupt this behavior. Our controls test that prediction by distinguishing ineffective paraphrase and appearance variation from effective same-instruction action alternatives. Identifying the missing supervision is what makes ECT simple. Once the diagnosis yields an effective remedy, added complexity is not a scientific contribution in itself. Its straightforward construction is a consequence of that insight, not a reason to discount it.

Reviewer_1ixaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper first investigates how instruction following in VLA models depends on the wording of the instruction and the scene in which it is executed. Based on this analysis, the authors introduce Equivariant Counterfactual Training (ECT), a method intended to improve model adaptation to new tasks. The proposed approach is evaluated on two VLA models: pi0.5 and GR00T-N1.7.

Critical strengths

Investigating the behavior of VLA models and identifying effective training strategies and data configurations is critically important. From this perspective, the paper addresses a relevant and timely research problem.

Critical weaknesses
  1. The paper is difficult to follow and lacks a coherent and comprehensive description of the experiments.

  2. Investigating how VLA model behavior changes in response to variations in instructions and/or environments is an active research direction [1, 2, 3]. From this perspective, the paper addresses important research questions. However, the proposed method is not sufficiently positioned against simpler approaches. Some form of scene randomization is standard practice when training such models, including varying the positions of target objects, introducing distractor objects, modifying object properties, textures, colors, and other scene attributes [4]. Therefore, it is not surprising that training on a single scene yields worse results than training on multiple scenes (e.g., lines 429–431: “The UR5e study allocates 400 demonstrations either to repetitions in one layout or to four layouts that require different reaches under the same instruction”). Given this experimental setup, it remains unclear what drives the observed improvements: scene diversity itself or the specific method used to generate this diversity. The authors should conduct more comprehensive ablation studies to disentangle these factors, particularly regarding the variability of the training data. For example, an additional baseline could involve training on scenes with randomly positioned objects within a predefined workspace. Another important limitation concerns the applicability of the proposed method. Not all real-world tasks allow the construction of additional scenes by mirroring the current object configurations.

Considering the above concerns, I believe that the paper is not yet ready for publication in its current form.

  1. Pugacheva, Daria, et al. "Bring the apple, not the sofa: Impact of irrelevant context in embodied ai commands on vla models." arXiv preprint arXiv:2510.07067 (2025).
  2. Shodiev, Damir, et al. "VLA Grounder: Language-Conditioning Space Optimization for Black-Box VLA Models." arXiv preprint arXiv:2607.04517 (2026).
  3. Yin, Zhaokai, and Zhipeng Zhang. "Grounded semantic re-binding for robust instruction generalization in vision-language-action models." arXiv preprint arXiv:2608.02497 (2026).
  4. Liu, Jijia, et al. "What can rl bring to vla generalization? an empirical study." Advances in Neural Information Processing Systems 38 (2026): 97121-97151.
Questions for authors

No questions

1 comment

We thank the reviewer for the feedback. In short, scene diversity alone does not help. Only variation that changes the required action under the same instruction does, as our diagnosis predicts.

  • Simpler approaches, on the same demos and training steps. Instruction and appearance randomization give nothing. Random placement helps because it changes the required action. Mirroring helps most:
π0.5, LIBERO-PRO (4-suite mean)SwapTask
Standard36.321.5
+ instruction augmentation (paraphrases)37.318.6
+ appearance randomization (textures, lighting)32.516.6
+ ECT via random object placement (suggested baseline)48.627.1
+ ECT via mirroring56.129.7
  • Why. Policies pick a familiar trajectory by instruction and use vision only to track it (Sec. 3, 4). Only action-changing variation breaks this binding.

  • [4] and UR5e. [4] randomizes a single object, so the instruction never has to choose between alternatives. Its multi-object split still drops 19 points after RL, while ECT's gain survives the same PPO post-training (Appendix G.5).

    UR5e shows which diversity matters. With the same 400 demos, four layouts that each require a different reach under one instruction raise success on positions none of them covers, from 3/40 to 35/40 (Table 4). Appearance diversity of the same size gives nothing (table above).

  • Mirroring is not required. We chose it as the cheapest valid construction, since the action transforms exactly with the scene and no new demos are needed. The principle is the supervision (Eq. 5), and any construction that provides it works, including random placement (above), natural counterparts in CALVIN, and physical layouts on UR5e.

  • [1]–[3] vary wording only, so the action is unchanged. Paraphrase robustness does not fix our failure, since π0.5 already scores 95.6 on paraphrase yet 27.1 on Swap, and training on paraphrases leaves Swap unchanged (table above). Concurrent work [3] finds the right task kept internally but executed poorly under paraphrase. We find the opposite. Execution is fine, but the wrong, familiar task is selected (Sec. 4).

  • Presentation. We will add one overview table of all experiments (model, data, protocol) and polish the writing.

We would be grateful if the reviewer reconsidered their assessment of our contributions given these clarifications and results. We warmly invite further discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers for finding the problem "relevant and timely" (1ixa), the paper "logically structured" with "comprehensive" evidence across simulation, public datasets and real robots (CvLk), and the diagnosis "substantive... beyond aggregate success rates" (EP2L). Our paper follows one line of reasoning, which prior VLA generalization work has missed at each step:

  1. Benchmarks hide the failure. High LIBERO-PRO Semantic and Object scores are read as language and object understanding, and low Swap/Task as weak generalization. The split actually follows whether the required action changes (Sec. 2). Paraphrase and appearance changes keep the familiar trajectory valid, so replaying it still succeeds. Only Swap and Task need a different action, and only there do models fail (Sec. 3). Models do read language and see objects (Sec. 4), but nuisance scores cannot show whether they use them to select the action.
  2. The failure is instruction-action binding. Prior work says language is ignored or too weak to override vision. In fact both are used: language picks which familiar trajectory to run, and vision only guides its execution (Sec. 3). 15% of rollouts switch to another demonstrated task, and intervening on task-conditioned states redirects actions (Sec. 4).
  3. The cause is supervision. When demos pair each instruction with one narrow action family, as in LIBERO, an instruction-keyed policy fits them (Sec. 2.1).
  4. ECT supplies the missing supervision. Same instruction, distinguishable scene, different required action (Eq. 5). Only such action-changing variation helps: 4× more data (Swap/Task stay ≥34 points below ID), appearance randomization and instruction augmentation do not.

Revision plan. We will (i) state this reinterpretation of LIBERO-PRO in the Abstract and Introduction, (ii) define ECT by its supervision (Eq. 5), with mirroring as one construction, (iii) contrast prior diagnoses and remedies in related work, adding LIBERO-Gen and the cited works, (iv) present balanced sampling as the main part of the ECT loss (Fig. 3, Sec. 5.1), and (v) add the new experiments.

We warmly invite the reviewers to discuss and are glad to expand on any point.

AuthorsOpenReview ↗

Clarifying Our Core Contribution: Diagnosis and Missing Supervision

We thank the reviewers and the AC. The reviewers raised three main concerns, and our responses answer each with evidence.

  • Is the gain just scene diversity? No. With data and steps matched, paraphrase and appearance randomization provide little or no improvement in counterfactual success. Only same-instruction alternatives that change the required action improve it.
  • Is the failure already known, or a generic shortcut? Previous work attributes these failures to models ignoring either language or key visual cues. LIBERO-Gen studies failures to represent task goals or execute new task combinations. We identify which familiar task the model executes instead. Language selects it while vision tracks its shifted target despite the wrong task choice. Readouts reflect this choice, and state interventions redirect behavior. Narrow per-instruction action support lets this shortcut and grounded policies fit the same demonstrations. Distinguishable scenes requiring different actions under the same instruction provide the supervision needed to separate them.
  • Is the method too straightforward? Only in hindsight. Our diagnosis predicted which supervision helps, and the controls above confirm it. The substantial gains from this simple intervention demonstrate the value of the diagnosis without requiring a more complicated method. Deriving an effective remedy from a precise diagnosis is the strength of this work.

These results establish a diagnosable failure that targeted training supervision addresses, without a more complex method. We will make these distinctions more explicit in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.