- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies instruction–action binding in vision-language-action models: instructions can trigger familiar task behaviors, while visual feedback adjusts execution, without correctly selecting the action required by the current instruction and scene.
Its contributions include behavioral and internal-state analyses of this failure in and GR00T-N1.7, an explanation based on insufficient scene-dependent action diversity in demonstrations, and Equivariant Counterfactual Training (ECT). ECT supplies valid demonstrations where the same instruction requires different actions across scenes and trains corresponding examples together using standard imitation losses.
Experiments on LIBERO-PRO, CALVIN, and a real robot show improved generalization, particularly under position changes. Most gains come from counterfactual data, while paired training provides additional but inconsistent benefits.
Critical strengths
-
The paper provides a substantive diagnosis beyond aggregate success rates. Behavioral probes and internal-state analyses show that language sensitivity and visual tracking can coexist with incorrect task selection. This distinction is consequential because it challenges the use of modality responsiveness as evidence of grounded instruction following and identifies a concrete failure mode for evaluation and training.
-
The supervision principle receives complementary empirical support. Controlled LIBERO-PRO experiments separate the contributions of counterfactual data and paired training, while CALVIN demonstrates benefits from pairing existing demonstrations and the real-robot study shows gains under a fixed demonstration budget. Together, these results support the practical importance of scene-dependent action alternatives beyond simply collecting more demonstrations.
Critical weaknesses
- The incremental contribution over closely related work is insufficiently established. Prior work already documents familiar-task execution under counterfactual instructions and proposes counterfactual supervision or geometric demonstration augmentation. Although the paper offers a more nuanced diagnosis, its experiments do not clearly distinguish instruction–action binding from existing shortcut-learning explanations or directly compare ECT with closely related methods. This makes it difficult to assess whether the proposed framework yields substantially new explanatory or methodological value.
- The benefit of paired training is not sufficiently isolated. Most LIBERO-PRO gains come from counterfactual data, while pairing provides smaller, inconsistent improvements. The paired setup jointly changes counterpart co-presentation, original/counterfactual sampling balance, and shared flow-matching noise and time. Consequently, the experiments do not establish which ingredient drives the additional gains. Since paired training is a central methodological contribution, this attribution gap materially weakens the evidence for its novelty and effectiveness.
Questions for authors
- How does ECT compare directly with existing demonstration-augmentation methods, such as RoCoDA, under matched data and training budgets? Without this comparison, how can we distinguish the benefits of ECT’s specific design from those of broader scene and action coverage?
- Why does the third block of Table 2 report only ECT data results for GR00T-N1.7, without ECT loss? Was paired training evaluated but found ineffective, or was it omitted for other reasons? This distinction matters for assessing whether the full method generalizes across architectures.
3 comments
I thank the authors for the additional experiments. They clarify that most gains come from counterfactual data and balanced sampling, while the independent benefit of pairing remains unestablished. My main concern about the methodological contribution therefore remains, and I retain my original rating.
We thank the reviewer for the initial question of whether the gain comes from pairing. Our submission left the mechanism of the ECT loss as an open question, and the reviewer’s question turned it into a concrete result, which we appreciate. In controlled experiments, balanced sampling is the main source of the additional gain. ECT already balances originals and counterparts through matched-pair sampling, so this clarifies an existing component, not a new method. We will revise Sec. 5.1 and Fig. 3 accordingly and no longer claim a pairing-specific contribution. The controls resolve the attribution concern. Our central contribution remains the diagnosis of instruction-action binding and the supervision principle it reveals. We respectfully ask the reviewer to reconsider their assessment and to specify which concerns remain unresolved.
We thank the reviewer for the specific questions, which we answer with new controlled experiments. In short, benchmark scores hide a failure in which language selects a familiar trajectory and vision only tracks it. We trace it to supervision, and ECT supplies the missing supervision (see also the General Response).
W1, distinguishing binding from shortcut explanations. Binding is a specific shortcut that ignored-language and vision-prior accounts do not explain. Both modalities are active. With the target swapped and shifted by 10 cm, the arm follows the object in the source-bound slot, not the named one (Sec. 3), and in LIBERO-Goal 37% of rollouts switch to another demonstrated task. Training permits this, because an instruction-keyed policy fits demos in which each instruction covers a narrow action family (Sec. 2.1). Unlike CAG/CAST (instruction varied, scene fixed) and RoCoDA (no language), ECT fixes the instruction and changes the required action (Sec. 6).
The choice is also visible inside the model. Reading π0.5's internal state identifies the wrongly executed task beyond word overlap, and injecting another task's state redirects the arm in both models (Sec. 4):
W2, isolating co-presentation, sampling balance and shared noise/time. With the data fixed, balance explains most of the additional ECT loss gain on both benchmarks, and exact pairing adds a smaller gain on top. This sharpens our account. ECT data supply missing same-instruction action alternatives, and the ECT loss balances existing ones. Both implement the supervision principle derived from our diagnosis.
We thank the reviewer for this question, which makes the paper clearer, and will revise Sec. 5.1 and Fig. 3 to present the ECT loss as balanced counterpart sampling. Matched updates, steps and data pool:
Q1, RoCoDA under matched data and training budget. Swap means at 30K steps are 32.5 for action-invariant randomization, 48.6 for RoCoDA-style random-pose placement, and 56.1 for ECT data via mirroring.
Q2. Due to compute limits, paired GR00T training was not evaluated at submission. We now report it (3 seeds).
As on π0.5 (Table 3), the LIBERO gain comes mainly from ECT data. The ECT loss shows its value where counterparts already exist, raising CALVIN five-task completion from 58% to 76% with no new demos (Table 4). GR00T results:
We hope this answers the reviewer's two central questions, namely what binding adds beyond generic shortcut accounts (which shortcut, how internal states drive it, why imitation permits it) and what each ECT-loss ingredient contributes. We would be grateful if the reviewer reconsidered their assessment of our contributions given these clarifications and results. We welcome further engagement.