acceptodds
Under review as a conference paper at ICLR 2027

ActRight: Counterfactual Training for Causally-Grounded Vision-Language-Action Models

Abstract

Vision–Language–Action (VLA) models achieve strong performance in robotic manipulation, but existing approaches optimize task success without ensuring that actions are grounded in language semantics. As a result, models often rely on spurious visual correlations, leading to failures under distribution shifts and novel vision–language compositions. We propose ActRight, a training framework that enforces counterfactual causal constraints for cross-modal decision making, with three key technical contributions. First, we impose a structured V→L→A informa- tion flow by introducing an intermediate language representation as the sole input to the policy. Second, we introduce counterfactual objectives that constrain model behavior under interventions on vision and language, enforcing uncertainty under mismatched inputs and sensitivity to changes in language semantics. Third, we de- velop an adversarial mechanism that actively constructs challenging counterfactual scenarios to expose and penalize violations of causal consistency during training. Experiments on the LIBERO and CALVIN benchmarks show that ActRight consis- tently outperforms existing VLA methods in robustness and generalization, while improving sensitivity to language conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XeAEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ActRight aims to reduce visual shortcut learning in vision-language-action models. It introduces an intermediate representation ll, combining vision, language, and proprioception, with instruction reconstruction used to preserve language information. Counterfactual objectives encourage uncertainty under shuffled observations and different action predictions under shuffled instructions. An adversarial generator supplies additional perturbations. Experiments report high success on LIBERO and CALVIN, with component ablations suggesting that the proposed training scheme improves task performance.

Critical strengths

The ablations provide useful evidence that the components help task performance. Removing the architecture reduces average LIBERO success from 97.6% to 95.4%, and removing counterfactual rewards reduces it to 94.6% (Table 3). These results support the method's practical potential, even though they do not establish the proposed causal explanation.

Critical weaknesses
  1. The central grounding claim is not convincingly supported. Reconstructing instructions from the intermediate representation ll does not prevent visual shortcuts from being stored alongside language information. Shuffled inputs are not necessarily incompatible, and different action predictions need not be correct ones (Section 3.3). The randomized-representation ablation could also reflect ordinary dependence on a learned representation. The announced grounding evaluations would help distinguish these explanations, but their results are not reported.

  2. The optimization and evaluation contain unresolved inconsistencies. Equation (11) maximizes the adversarial objective, V\mathcal{V}, while the text below Eq. (13) says it is minimized. The distance penalty also appears with a sign that rewards larger perturbations under maximization. In the experiments, CALVIN's “Avg. len = 96.9” has an unclear definition, and the unified LIBERO results average to 97.95%, not the reported 97.2% (Tables 1–2). These issues make the method and its gains difficult to assess reliably.

Questions for authors
  1. How is the intermediate representation ll randomized in the ablation? How often do shuffled pairs genuinely mismatch? Could you provide the announced grounding evaluations, including a small test with feasible alternative instructions in the same scene?

  2. What losses were actually used for the policy and generator? Please reconcile the optimization directions and distance-penalty sign, and clarify which terms pass through gradient reversal.

  3. What does CALVIN's “Avg. len = 96.9” measure? Which chain length does 99.5% describe, and do the baseline entries use the same definitions?

  4. Could you resolve the LIBERO averaging discrepancy and report evaluation counts and variation across training runs?

1 comment

Thank you for the thorough review.

[W1] Grounding claim and missing evaluations. We now report them. With the same scene and a swapped feasible instruction, ActRight completes the new task in 87.1% of episodes, vs 65.6% and 34.2% (R1). Without an instruction, it succeeds on only 4.7–11.8% of episodes, vs up to 40.3% for the baselines (R1). Under task-irrelevant visual changes, its average drop is 3.1 pts, vs 7.7 and 11.3 (R5). We agree that reconstruction and the randomized-ll ablation alone do not establish grounding, and we have revised the claims to rest on E1 and E5 (C4). For shuffled pairs, ~10% share a task (R4). These give zero KL for Ωl\Omega_l and act as bounded noise for Ωv\Omega_v. We will add task-ID filtering.

[W2] Inconsistencies.

  1. Optimization: Eq. (14) is the implemented loss, with C\mathcal C computed before the GRL; the explicit objective is min⁡θ,ϕmax⁡ω V−η C\min_{\theta,\phi}\max_\omega\ \mathcal V-\eta\,\mathcal C, so the proximity term penalizes large perturbations (C1).
  2. CALVIN: 96.9 is p2p_2; Avg. len is 4.59 (R3).
  3. LIBERO: the correct average is 97.95%.

[Q1] Randomized ll and mismatch rate. The randomization is applied at test time only, on the frozen full model. For each sample, the complete mediator representation is replaced with that of a sample from a different task: lirand=lσ(i)l_i^{\mathrm{rand}}=l_{\sigma(i)} with σ(i)≠i\sigma(i)\neq i and task(σ(i))≠task(i)\mathrm{task}(\sigma(i))\neq\mathrm{task}(i), using a within-batch permutation over parallel environments and a pool of other-task representations when the batch has none. The environment, instruction, and success criterion are unchanged. Unlike Gaussian noise, this preserves the marginal distribution and scale of ll. The resulting drop (97.6 → 92.3) shows that the policy depends on ll, but not what ll encodes, which E1 addresses. About 90% of within-batch training pairs come from different tasks (R4).

[Q2] Losses and GRL. With α=0.1\alpha=0.1, β=1.0\beta=1.0, and η=0.01\eta=0.01, the stage-2 loss is

L=Ltask+0.1 Lrec−0.1 Ωv−0.1 Ωl+0.05 [ VGRL+0.01 C ],V=−H−DKL.\mathcal L=\mathcal L_{\mathrm{task}}+0.1\,\mathcal L_{\mathrm{rec}}-0.1\,\Omega_v-0.1\,\Omega_l+0.05\,[\,\mathcal V_{\mathrm{GRL}}+0.01\,\mathcal C\,],\qquad \mathcal V=-\mathcal H-D_{\mathrm{KL}}.

The policy and mediator receive the ordinary gradient of V\mathcal V, so they minimize violations. The violation branch reaches the generator through the GRL (coefficient 1), while C\mathcal C is computed from the generator outputs before the GRL (not detached) and back-propagates normally. The generator therefore minimizes −V+η C-\mathcal V+\eta\,\mathcal C, i.e., it seeks harder counterfactuals while staying close to the original features. The counterfactual and adversarial terms are applied on a fraction ρ=0.3\rho=0.3 of batches. The reconstruction and distance terms use the per-sample squared L2 norm averaged over the batch.

[Q3] CALVIN. 99.5 and 96.9 are p1p_1 and p2p_2. R3 reports p1p_1–p5p_5 and Avg. len =∑kpk=\sum_k p_k (range 0–5) for all methods under the official 1,000-chain protocol.

[Q4] Averaging, counts, and runs. The unified average is 97.95%. The new evaluations use 1,000 episodes per suite and condition, 1,800 for the Goal swap test, and 1,000 CALVIN chains. Main results come from one training run per setting because of the cost of training 7B VLA models (about 20 h on 8×A100 per run). The new results agree in direction across all suites and all independent evaluations.

We are happy to provide further details before the discussion closes.

Reviewer_2B7KOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses language neglect in vision-language-action policies. ActRight introduces a continuous mediator that fuses visual, language, and proprioceptive features, reconstructs the instruction representation, and supplies the policy's visual input slots. Training combines action supervision with entropy maximization under visual substitutions, policy-divergence maximization under language substitutions, and an adversarial counterfactual generator. Results are reported on LIBERO and CALVIN, with component ablations on LIBERO.

Critical strengths

The motivation is important: successful execution alone does not establish that a robot follows the instruction rather than exploiting scene correlations. Studying behavior under modality changes is therefore worthwhile. The reported LIBERO success rates are competitive, and the component ablations suggest that the complete training recipe has an empirical effect beyond the mediator alone.

Critical weaknesses

The architecture does not establish the claimed semantic mediation. In Eq. 1, l is an unconstrained continuous fusion of visual, language, and proprioceptive features. Restricting the policy to this representation imposes a computational bottleneck, but the bottleneck can retain visual shortcuts. For example, l could encode the instruction in one subspace for reconstruction and visual cues in another subspace used by the action head. Neither instruction reconstruction nor a performance drop after randomizing l rules this out. The paper needs to distinguish dependence on a learned latent from dependence on instruction semantics. Besides, the counterfactual objectives do not specify correct counterfactual behavior. A within-batch visual substitution does not necessarily make an instruction ungroundable, yet the visual objective always encourages uncertainty. Likewise, maximizing action-distribution KL after replacing the instruction rewards any change, including an incorrect response or unnecessary sensitivity when two instructions permit the same immediate action. These losses may be useful regularizers, but the inference from uncertainty/sensitivity to causal correctness is not established. The bounded feature perturbations also do not establish that the generated examples implement meaningful semantic interventions.

The evaluation does not directly test the central claim. Section 4.1 announces language sensitivity, visual robustness, and unseen vision-language compositions, but the presented results consist of aggregate benchmark success and component ablations. These cannot distinguish improved causal grounding from a generic training or representation effect. The decisive evaluation would hold the scene fixed while changing to another feasible instruction and measure success on the newly requested task, together with action invariance under irrelevant scene changes. This is more central than further improving an already high aggregate success rate.

Questions for authors

What excludes a representation that reconstructs the instruction but drives actions from a separate visual shortcut? Can you provide a behavioral test of this distinction?

How do you determine that a substituted scene invalidates the instruction, and how do you check that a response to a changed instruction is correct rather than merely different?

Which generator update is implemented, and what benefit does the adversarial generator add beyond the non-adversarial counterfactual losses?

Please clarify the CALVIN metrics and their units, especially the 96.9 entry labeled Avg. len, and reconcile the unified-policy LIBERO average with its four component entries.

2 comments

Thank you for the expert review. Your key point, distinguishing dependence on a learned latent from dependence on instruction semantics, shaped E1 and E5, which implement the two tests you proposed.

[W3, Q1] Direct test of the central claim.

  • Same scene, new feasible instruction (R1): ActRight completes the newly requested task in 87.1% of 1,800 episodes, vs 65.6% (OpenVLA-OFT) and 34.2% (OpenVLA), although OpenVLA-OFT is slightly higher on the original instructions (97.8 vs 96.7).
  • No instruction (R1): ActRight's SR on the original task falls to 4.7–11.8% across suites, vs up to 40.3% (OpenVLA) and 32.1% (OpenVLA-OFT): the baselines often solve tasks from the scene alone.
  • Irrelevant scene changes (R5): ActRight's average drop is 3.1 pts, vs 7.7 and 11.3.

ActRight is thus both more sensitive to the instruction and more invariant to task-irrelevant visual changes, which a generic training effect would not easily explain.

[W1, Q1] Instruction in one subspace, visual shortcut in another. We agree that Eq. (1) plus reconstruction does not exclude this by construction, and we have revised the claims (C4). E1 is the behavioral test: the scene and initial state, and hence the visual input to the mediator, are held fixed, and only the instruction changes. A policy driven by a visual subspace would not redirect to the new goal; ActRight does so in 87.1% of episodes. This does not reveal how ll encodes information internally, and the revised Limitations state this explicitly.

[W2, Q2] Correct counterfactual behavior. We agree; the revised Section 3.3 presents Ωv\Omega_v and Ωl\Omega_l as regularizers encoding necessary conditions, with correctness assessed behaviorally.

  • Scene substitutions: ~10% of pairs share a task (R4), which is bounded label noise for Ωv\Omega_v. For Ωl\Omega_l, identical instructions give zero KL. We will add task-ID filtering.
  • "Different" vs "correct": E1 scores the new task's goal predicate, not a change in the action distribution.
  • Feature perturbations: the generator acts as hard-example mining within a δmax⁡\delta_{\max} ball, not as a semantic intervention, and the revised Section 3.4 now describes it as such.

[Q3] Generator update and its benefit. See (C1) for the update. The benefit is +1.0 pt on average and positive on all four suites (R2).

[Q4] CALVIN and LIBERO. 99.5 and 96.9 are p1p_1 and p2p_2; the full results are in R3 (Avg. len 4.59). The unified-policy average is 97.95%.

We hope these results directly address your concerns and would be grateful if you would take them into account.

Reviewer_2B7KOpenReview ↗

I maintain my original rating

I maintain my original rating for the following reasons:

  1. Page limit violation. The conclusion begins on page 10, which means the main text exceeds the allowed page limit stated in the submission guidelines. This is an objective formatting violation, independent of the scientific content.
  2. Figure quality. The quality of Figures 1 and 2 is very low: they are generic schematic illustrations rather than figures derived from the actual implementation, and they lack the concrete detail (specific configurations, data sources, implementation parameters) needed to support the claims in the text. In their current form they do not document a serious, completed study, and the proposed method cannot be verified from them.

Given the formatting violation and the insufficient technical substance, I do not believe the paper meets the bar for acceptance in its current form.

Reviewer_gFR6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ActRight, a training framework aimed at improving language grounding in vision-language-action models and reducing reliance on spurious visual correlations. It introduces three components: a language-mediated architecture that routes visual information through an intermediate representation before action prediction; counterfactual objectives that encourage uncertainty under vision–language mismatches and sensitivity to instruction changes; and an adversarial generator that constructs challenging counterfactual inputs during training. The authors evaluate ActRight on LIBERO and CALVIN, reporting improved task success relative to existing VLA baselines, and provide ablation studies examining the architectural constraint and counterfactual objectives.

Critical strengths
  1. The paper is well written and easy to follow. The motivation, method, and experiments connect logically, making it clear what the authors propose and how they test it.

  2. The paper focuses on whether policies actually follow language instructions rather than exploit visual correlations. This distinction is consequential because high benchmark success alone may conceal failures under instruction changes, making language grounding a meaningful target for improving policy reliability.

  3. The framework constructs counterfactual inputs through within-batch permutations and computes its objectives over action-token distributions without full autoregressive rollouts. This offers a potentially practical way to encourage language-dependent behavior using existing demonstration datasets.

Critical weaknesses
  1. The incremental benefit of adaptive counterfactual generation is not established. Table 3 removes all counterfactual supervision or individual modality objectives, but does not retain the architecture and permutation-based objectives while disabling the adversarial branch. Since adaptive generation is presented as a principal contribution, a controlled comparison—for example, setting λadv=0\lambda_{\mathrm{adv}}=0 while retaining λv\lambda_v and λl\lambda_l—would clarify what it adds beyond within-batch permutations.
Questions for authors
  1. Could the authors clarify the generator objective and gradient routing? In Eq. (11), maximizing +ηC+\eta C for positive η\eta encourages larger perturbations, although CC is defined as squared distance and described as enforcing proximity. Should the generator objective instead include −ηC-\eta C, and does CC bypass gradient reversal in the implementation? Additionally, the text following Eq. (13) states that the generator minimizes V\mathcal V, whereas Eq. (11) and Algorithm 1 specify maximization. Please reconcile these descriptions with the implemented updates.

  2. Could the authors clarify how counterfactual pairs are selected in Section 3.3? A batch-permuted observation may still support the original instruction, and a permuted instruction may be identical or semantically equivalent to the original. Does batch construction or task-/instruction-level filtering exclude such cases? Please explain how potentially compatible or equivalent pairs are handled when applying the entropy and KL objectives.

1 comment

Thank you for the positive review. Results are in the General Response.

[W1] Benefit of adaptive generation. We ran the requested control (R2): λadv=0\lambda_{\mathrm{adv}}=0 with λv,λl\lambda_v,\lambda_l, the permutation-based objectives, reconstruction, and two-stage training unchanged. The average SR drops from 97.6 to 96.6, and the drop appears on every suite (largest on Long, −1.6, and Goal, −1.1). Without any counterfactual terms the average is 94.6. Thus the permutation-based objectives give +2.0 pts, and adaptive generation adds a further +1.0 pt.

[Q1] Generator objective and gradient routing. See (C1). Eq. (14) is the implemented loss, and C\mathcal C is computed before the GRL, so it bypasses reversal. The generator receives −∇ωV+η∇ωC-\nabla_\omega\mathcal V+\eta\nabla_\omega\mathcal C, i.e., it maximizes V−η C\mathcal V-\eta\,\mathcal C, while the policy minimizes V\mathcal V. Eq. (11) and the text after Eq. (13) have been rewritten in this explicit form in the revised PDF, and Algorithm 1 now shows the single joint update.

[Q2] Compatible or equivalent pairs. Pairs come from a within-batch cyclic shift without task filtering. About 10% are same-task (hence same-instruction) pairs (R4).

  • Language branch: identical instructions give l~(l)=l\tilde l^{(l)}=l and DKL=0D_{\mathrm{KL}}=0, so these pairs contribute no gradient. The policy is never pushed to respond differently to an equivalent instruction.
  • Visual branch: same-task swaps are label noise for Ωv\Omega_v, bounded by ρ=0.3\rho=0.3 and λv=0.1\lambda_v=0.1.

We will add task-ID filtering to the sampler. E1 further checks that the response to a changed instruction is correct (R1: 87.1% new-task SR).

We are happy to answer any follow-up questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary and clarifications

We thank all reviewers for their careful reviews. They agree that the problem matters, i.e., whether a VLA policy follows the instruction rather than exploiting scene correlations (gFR6, 2B7K, XeAE), and that each component has an empirical effect (2B7K, XeAE). The main concerns were (i) the lack of a direct grounding test, (ii) the missing generator ablation, (iii) the validity of within-batch pairs, and (iv) presentation issues in equations and tables. We add five experiments (E1–E5, results in Part 2) and the clarifications below.

IDExperimentKey result
E1Same scene, swapped feasible instruction (LIBERO-Goal); empty/irrelevant instruction (all suites)New-task SR 87.1% vs 65.6% (OpenVLA-OFT) / 34.2% (OpenVLA)
E2λadv=0\lambda_{\mathrm{adv}}=0 ablation97.6 → 96.6, lower on all suites
E3CALVIN ABC→D, official protocolAvg. len 4.59
E4Within-batch pair statistics~10% same-task pairs
E5Task-irrelevant visual perturbationsAvg. drop 3.1 vs 7.7 / 11.3 pts

(C1) Generator objective. Eq. (14) is the implemented loss: Ladv=V+η C\mathcal L_{\mathrm{adv}}=\mathcal V+\eta\,\mathcal C is minimized over all parameters, with a gradient reversal layer (GRL) between the generator outputs and the mediator, and C\mathcal C computed before the GRL. Eq. (11) and the sentence after Eq. (13) used an imprecise shorthand. Written explicitly, the objective is min⁡θ,ϕmax⁡ω V−η C\min_{\theta,\phi}\max_\omega\ \mathcal V-\eta\,\mathcal C: the policy minimizes violations, while the generator maximizes them under the proximity penalty, with an additional hard bound ∥δ∥∞≤δmax⁡\|\delta\|_\infty\le\delta_{\max} from the tanh⁡\tanh. In the revised PDF, Eq. (11) is rewritten in this form, and the generator is renamed GωG_\omega to avoid the clash with the decoder gψg_\psi.

(C2) CALVIN. In Table 2, 99.5 and 96.9 are p1p_1 and p2p_2, not Avg. len; the column was mislabeled. The full metrics are in E3.

(C3) LIBERO unified average. The correct mean of 97.4 / 99.2 / 97.7 / 97.5 is 97.95% (not 97.2%).

(C4) Scope of claims. We agree that the mediator is a structural bottleneck and does not by itself guarantee semantic mediation, and that uncertainty and sensitivity are necessary rather than sufficient conditions for grounding. In the revised PDF, we replace "causally grounded" with "language-grounded" in the text, present the counterfactual terms as regularizers, and base the grounding claim on E1 and E5 (Limitations updated accordingly).

A revised PDF has been uploaded. All changes are highlighted in blue; the new results are in Appendix A–B, and the λadv=0\lambda_{\mathrm{adv}}=0 ablation is added to Table 3.

AuthorsOpenReview ↗

Summary and clarifications

R1 (E1). LIBERO-Goal, same scene. All ten Goal tasks share one scene. From the initial state of task ii, the policy receives the instruction of task jj, and success is judged by task jj's goal predicate. We use 90 feasible directed pairs × 20 episodes (1,800 episodes). Empty and irrelevant instructions are judged against the original task (↓ is better).

MethodOriginalSwapped: new task ↑Empty ↓Irrelevant ↓
OpenVLA79.434.224.823.5
OpenVLA-OFT97.865.618.617.2
ActRight96.787.18.97.6

Empty-instruction SR on Spatial / Object / Goal / Long (1,000 episodes per suite): OpenVLA 40.3 / 33.7 / 24.8 / 12.4; OpenVLA-OFT 32.1 / 26.4 / 18.6 / 8.7; ActRight 11.8 / 10.6 / 8.9 / 4.7.

R2 (E2). λadv=0\lambda_{\mathrm{adv}}=0 (all else unchanged; 1,000 episodes per suite).

VariantSpatialObjectGoalLongAvg
Full99.199.496.894.997.6
λadv=0\lambda_{\mathrm{adv}}=098.698.895.793.396.6
No counterfactual terms (Table 3)97.297.993.689.894.6

R3 (E3). CALVIN ABC→D, 1,000 chains; pkp_k = % of chains with the first kk instructions completed; Avg. len =∑kpk=\sum_k p_k.

Methodp1p_1p2p_2p3p_3p4p_4p5p_5Avg. len
OpenVLA91.377.862.052.143.53.27
OpenVLA-OFT96.992.085.780.472.94.28
Seer-Large96.391.686.180.374.04.28
UnifiedVLA98.994.889.082.875.14.41
VLA-Adapter99.194.688.882.876.54.42
FLOWER99.495.890.784.977.84.53*
ActRight99.596.992.887.482.64.59

*As reported; the per-step rates sum to 4.49.

R4 (E4). Within-batch pairs (training sampler, 64,000 pairs per suite). Same-task (= same-instruction) pairs: Spatial 10.2%, Object 10.0%, Goal 9.9%, Long 10.1%. Pairs with pooled-instruction cosine similarity > 0.9: 28.4% / 17.1% / 23.6% / 14.8%.

R5 (E5). Task-irrelevant visual changes (instruction and goal fixed; mean over four suites; 1,000 episodes per suite and condition).

MethodCleanDistractorsBackgroundLightingAvg. drop ↓
OpenVLA76.563.466.865.311.3
OpenVLA-OFT97.188.290.689.47.7
ActRight97.694.195.394.03.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.