- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ActRight aims to reduce visual shortcut learning in vision-language-action models. It introduces an intermediate representation , combining vision, language, and proprioception, with instruction reconstruction used to preserve language information. Counterfactual objectives encourage uncertainty under shuffled observations and different action predictions under shuffled instructions. An adversarial generator supplies additional perturbations. Experiments report high success on LIBERO and CALVIN, with component ablations suggesting that the proposed training scheme improves task performance.
Critical strengths
The ablations provide useful evidence that the components help task performance. Removing the architecture reduces average LIBERO success from 97.6% to 95.4%, and removing counterfactual rewards reduces it to 94.6% (Table 3). These results support the method's practical potential, even though they do not establish the proposed causal explanation.
Critical weaknesses
-
The central grounding claim is not convincingly supported. Reconstructing instructions from the intermediate representation does not prevent visual shortcuts from being stored alongside language information. Shuffled inputs are not necessarily incompatible, and different action predictions need not be correct ones (Section 3.3). The randomized-representation ablation could also reflect ordinary dependence on a learned representation. The announced grounding evaluations would help distinguish these explanations, but their results are not reported.
-
The optimization and evaluation contain unresolved inconsistencies. Equation (11) maximizes the adversarial objective, , while the text below Eq. (13) says it is minimized. The distance penalty also appears with a sign that rewards larger perturbations under maximization. In the experiments, CALVIN's “Avg. len = 96.9” has an unclear definition, and the unified LIBERO results average to 97.95%, not the reported 97.2% (Tables 1–2). These issues make the method and its gains difficult to assess reliably.
Questions for authors
-
How is the intermediate representation randomized in the ablation? How often do shuffled pairs genuinely mismatch? Could you provide the announced grounding evaluations, including a small test with feasible alternative instructions in the same scene?
-
What losses were actually used for the policy and generator? Please reconcile the optimization directions and distance-penalty sign, and clarify which terms pass through gradient reversal.
-
What does CALVIN's “Avg. len = 96.9” measure? Which chain length does 99.5% describe, and do the baseline entries use the same definitions?
-
Could you resolve the LIBERO averaging discrepancy and report evaluation counts and variation across training runs?
Thank you for the thorough review.
[W1] Grounding claim and missing evaluations. We now report them. With the same scene and a swapped feasible instruction, ActRight completes the new task in 87.1% of episodes, vs 65.6% and 34.2% (R1). Without an instruction, it succeeds on only 4.7–11.8% of episodes, vs up to 40.3% for the baselines (R1). Under task-irrelevant visual changes, its average drop is 3.1 pts, vs 7.7 and 11.3 (R5). We agree that reconstruction and the randomized-l ablation alone do not establish grounding, and we have revised the claims to rest on E1 and E5 (C4). For shuffled pairs, ~10% share a task (R4). These give zero KL for Ωl and act as bounded noise for Ωv. We will add task-ID filtering.
[W2] Inconsistencies.
[Q1] Randomized l and mismatch rate. The randomization is applied at test time only, on the frozen full model. For each sample, the complete mediator representation is replaced with that of a sample from a different task: lirand=lσ(i) with σ(i)=i and task(σ(i))=task(i), using a within-batch permutation over parallel environments and a pool of other-task representations when the batch has none. The environment, instruction, and success criterion are unchanged. Unlike Gaussian noise, this preserves the marginal distribution and scale of l. The resulting drop (97.6 → 92.3) shows that the policy depends on l, but not what l encodes, which E1 addresses. About 90% of within-batch training pairs come from different tasks (R4).
[Q2] Losses and GRL. With α=0.1, β=1.0, and η=0.01, the stage-2 loss is
L=Ltask+0.1Lrec−0.1Ωv−0.1Ωl+0.05[VGRL+0.01C],V=−H−DKL.
The policy and mediator receive the ordinary gradient of V, so they minimize violations. The violation branch reaches the generator through the GRL (coefficient 1), while C is computed from the generator outputs before the GRL (not detached) and back-propagates normally. The generator therefore minimizes −V+ηC, i.e., it seeks harder counterfactuals while staying close to the original features. The counterfactual and adversarial terms are applied on a fraction ρ=0.3 of batches. The reconstruction and distance terms use the per-sample squared L2 norm averaged over the batch.
[Q3] CALVIN. 99.5 and 96.9 are p1 and p2. R3 reports p1–p5 and Avg. len =∑kpk (range 0–5) for all methods under the official 1,000-chain protocol.
[Q4] Averaging, counts, and runs. The unified average is 97.95%. The new evaluations use 1,000 episodes per suite and condition, 1,800 for the Goal swap test, and 1,000 CALVIN chains. Main results come from one training run per setting because of the cost of training 7B VLA models (about 20 h on 8×A100 per run). The new results agree in direction across all suites and all independent evaluations.
We are happy to provide further details before the discussion closes.