- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Summary
The paper argues that VLA policies often rely on modality shortcuts: cues that are reliable in successful demonstrations, such as wrist-view salience, the object noun, or the close-then-lift sequence, which replace the task evidence a decision should depend on. It contributes three things:
- An information-theoretic analysis of when such shortcuts arise.
- Perturbot, three training-data interventions:
- V: wrist-view distractor insertion;
- C: decision-relevant detailed captions;
- R: random and failed trajectories relabeled with their local behavior.
- GroundFscore (GF), an offline metric built from paired null and causal input edits.
With a π0.5 backbone, Perturbot raises real-robot General PnP success from 42% to 84% and improves RoboTwin 2.0 results.
Critical strengths
Strengths
-
Clear theoretical motivation. Sec. 3.1 unifies three seemingly different failures (salience capture, noun lock-in, motor inertia) under one information-theoretic view. It explains why more demonstrations of the same kind need not remove them (collinearity and degeneracy), and it derives two distinct data-level remedies: lowering redundancy and raising evidence informativeness.
-
GroundFscore is well designed.
- Combining stability (1 − R_c) and causal sensitivity (S_c) in a harmonic mean rules out degenerate solutions. A policy that ignores its input scores zero, and so does one that reacts to everything.
- Projecting the response onto the required change Δa* rewards only movement in the correct direction.
- It needs only paired forward passes with shared sampling noise, so it is cheap and model-agnostic.
- It predicts online failures much better than task success does (Spearman 0.72 vs. 0.31, Table 13).
An offline diagnostic of this kind is a useful contribution in its own right.
-
Thorough real-robot ablation. The full V/C/R factorial is run together with budget-matched mechanism controls.
Critical weaknesses
Weaknesses
W1. RoboTwin success gains do not show that shortcuts were reduced
The central claim is that Perturbot improves VLAs by reducing shortcuts. RoboTwin 2.0, however, is not designed to isolate shortcuts. Its failures can stem from manipulation precision, bimanual coordination, long horizons, or contact dynamics. A higher success rate therefore does not show that the improvement comes from reduced shortcut reliance rather than generic gains from extra data coverage (R) or richer supervision (C). GF, which the paper proposes for exactly this purpose, is not reported on RoboTwin.
Two issues further weaken the RoboTwin evidence:
- No matched baseline. The π0.5 row in Table 2 is the published number from Cai et al. (2026b), identical to Table 7, whereas the Perturbot rows use the authors' own fine-tuning recipe. The headline +12.2-point gain thus compares two different training recipes.
- Selective presentation. Main-text Table 2 lists only systems that Perturbot outperforms, and the text claims "the best randomized-scene result among the listed systems." Appendix Table 7 contains many stronger systems (see W2).
W2. Strong models outperform Perturbot without it; whether the method helps them is unknown
Several open-source models with released RoboTwin 2.0 checkpoints or post-training code outperform Perturbot by about 10 points without any shortcut-targeted augmentation:
- Full: LingBot-VA 92.2 and ABot-M0.5 94.1, vs. 83.8 for Perturbot;
- Clean2Random-Randomized: GigaBrain-0.7 67.9, vs. 58.2 for Perturbot. The paper itself calls this setting "the more direct test of modality shortcuts."
This suggests either that shortcuts are not the dominant failure source, or that strong models already have low shortcut reliance. In both cases the practical value of Perturbot is unclear. Since all experiments use only π0.5, I ask the authors to:
- report GF for one or two of these strong models, to show whether they suffer from shortcuts at all;
- apply Perturbot to at least one of them on RoboTwin.
If GF does not improve further, or success drops, the proposed augmentation scheme would be of limited use or would even conflict with what strong models already learn. The method's benefit may then be confined to compensating for a weaker baseline.
W3. A dedicated shortcut testbed is needed
To support the claim that the method works because it reduces shortcuts, the authors should evaluate on a setting designed to expose them, rather than relying on RoboTwin success rates. The existing probes (50-trial salience-capture and motor-inertia probes on the real robot, and 5 verb-change task pairs in simulation) are small. A controlled simulated testbed would be more convincing: vary the correlation between the shortcut cue and the task evidence in the training data, and test under cue–evidence conflict. If Perturbot targets shortcuts, its benefit, in both success and GF, should grow with the correlation strength and vanish without it.
W4. Limited methodological novelty
Each intervention closely resembles existing techniques:
- V: generative or semantic image augmentation;
- C: instruction relabeling or enrichment;
- R: hindsight relabeling of play data (Lynch et al., 2020).
The contribution therefore lies mainly in their combination rather than in new methodology. The paper also misses the most closely related work on shortcut learning in generalist robot policies: Xing et al., "Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation" (arXiv 2508.06426). That work also attributes shortcuts to dataset structure and mitigates them with targeted data augmentation, and the paper should discuss how its approach differs.
Questions for authors
N/A
6 comments
Response to Reviewer RbAn (2/4)
💬 W2: Do stronger models still exhibit shortcuts, and can they benefit from PerturBot?
We agree that results on alone do not establish the method's utility for stronger models. We therefore evaluate LingBot-VA [1] and ABot-M0.5 [2] under Full, and GigaBrain-0.7 [3] under Clean2Random (clean-only Co-Train). Within each backbone, we compare our reproduced RoboTwin baseline with the same training recipe augmented by PerturBot, matching initialization, base task demonstrations, and training budget.
Shortcut reliance and mitigation. All three reproduced baselines still exhibit salience capture, noun lock-in, and motor inertia. Adding PerturBot reduces every shortcut failure rate while improving GroundFscore and RoboTwin success (Table 3). Average success increases by 0.5, 0.5, and 1.3 points for LingBot-VA, ABot-M0.5, and GigaBrain-0.7.
Table 3. RoboTwin shortcut diagnostics. Each entry is Baseline → + PerturBot. All values are percentages except GF. SC/NL/MI denote salience capture, noun lock-in, and motor inertia; GF is the V/L/A mean. Failure definitions follow W1. Success is the average in Table 4. Comparisons are within each backbone.
| Backbone (protocol) | SC failure ↓ | NL failure ↓ | MI failure ↓ | Mean GF ↑ | RoboTwin success ↑ |
|---|---|---|---|---|---|
| LingBot-VA (Full) | 16 → 13 | 18 → 12 | 14 → 11 | 0.58 → 0.60 | 92.2 → 92.7 |
| ABot-M0.5 (Full) | 17 → 14 | 23 → 16 | 15 → 12 | 0.53 → 0.56 | 94.0 → 94.5 |
| GigaBrain-0.7 (Clean2Random) | 21 → 14 | 25 → 16 | 19 → 12 | 0.48 → 0.52 | 66.9 → 68.2 |
Nominal benchmark performance. Table 4 reports the published results, our reproduced baselines, and + PerturBot. Each backbone follows its corresponding training protocol, with clean, randomized, and average success reported separately. We compare PerturBot with our own baseline, rather than with published numbers or models trained under different protocols.
Table 4. RoboTwin success (%). Baseline denotes our reproduction of each model's RoboTwin training recipe. Reported results are from [1–3].
| Backbone (protocol) | Result / Condition | Clean | Randomized | Average |
|---|---|---|---|---|
| LingBot-VA (Full) | Reported [1] | 92.93 | 91.55 | 92.24* |
| Baseline | 92.6 | 91.8 | 92.2 | |
| + PerturBot | 93.2 | 92.2 | 92.7 | |
| ABot-M0.5 (Full) | Reported [2] | 94.00 | 94.20 | 94.10 |
| Baseline | 93.8 | 94.2 | 94.0 | |
| + PerturBot | 94.4 | 94.6 | 94.5 | |
| GigaBrain-0.7 (Clean2Random) | Reported [3] | 66.80 | 67.90 | 67.35 |
| Baseline | 66.4 | 67.4 | 66.9 | |
| + PerturBot | 67.3 | 69.1 | 68.2 |
LingBot-VA's 92.24 is computed from its reported Easy/Hard results. Bold highlights our method, not the best published score.
Relative to our reproduced baselines, PerturBot improves Full-average success by 0.5 points for both LingBot-VA and ABot-M0.5, and Clean2Random-randomized success by 1.7 points for GigaBrain-0.7. These nominal point gains are modest but accompany clearer improvements on the shortcut probes. Our claim concerns improved task-relevant evidence use, rather than a new benchmark ranking.
Real-robot validation. On General PnP, PerturBot raises task success by 16, 12, and 18 points and improves GroundFscore in every family and on average (Table 5).
Table 5. Real-robot General PnP. Entries are Baseline → + PerturBot. Task success is the success rate on the General PnP cases. GF$_V$, GF$_L$, and GF$_A$ are GroundFscore on held-out observations, and Mean GF is their average.
| Backbone | Task success ↑ | GF$_V$ ↑ | GF$_L$ ↑ | GF$_A$ ↑ | Mean GF ↑ |
|---|---|---|---|---|---|
| LingBot-VA | 58 → 74 | 0.41 → 0.56 | 0.34 → 0.51 | 0.36 → 0.47 | 0.37 → 0.51 |
| ABot-M0.5 | 64 → 76 | 0.44 → 0.57 | 0.39 → 0.53 | 0.38 → 0.48 | 0.40 → 0.53 |
| GigaBrain-0.7 | 52 → 70 | 0.35 → 0.53 | 0.30 → 0.49 | 0.29 → 0.45 | 0.31 → 0.49 |
Together, these results support PerturBot as complementary to stronger pretraining, rather than merely compensating for a weaker backbone. They motivate scaling decision-relevant annotations and behavioral coverage alongside data quantity, instead of assuming that scale alone eliminates shortcuts.
References
[1] Lin Li et al. Causal World Modeling for Robot Control. arXiv preprint arXiv:2601.21998, 2026. https://arxiv.org/abs/2601.21998
[2] Ronghan Chen et al. ABot-M0.5: Unified Mobility-and-Manipulation World Action Model. arXiv preprint arXiv:2607.00678, 2026. https://arxiv.org/abs/2607.00678
[3] GigaBrain Team et al. GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture. arXiv preprint arXiv:2608.15875, 2026. https://arxiv.org/abs/2608.15875
Response to Reviewer RbAn (3/4)
💬 W3: A dedicated shortcut testbed is needed to establish the mechanism.
We directly vary training cue–decision agreement, , while keeping cue and decision frequencies balanced. We evaluate three controlled real-robot testbeds:
- Vision: a task-irrelevant salient marker predicts the target’s side, while the instruction and object identity determine the correct target.
- Language: object identity predicts a habitual pick or push operation, while the instruction’s verb determines the required operation. Both operations have valid demonstrations for every object.
- Motor context: two observable post-grasp checkpoints, 8 cm apart with the same height and orientation, predict habitual transport or regrasp. The actual grasp outcome determines the correct continuation: transport when holding the object, and regrasp when empty. All four checkpoint–outcome combinations have valid expert continuations. We evaluate continuation choices, not identical continuous actions across checkpoints.
Each setting uses 400 base demonstrations, three training seeds, and 50 held-out layouts evaluated under all four cue–evidence combinations. Methods share initialization, base task data, and optimization budget. The matched-data control (MDC) also shares PerturBot’s auxiliary data but uses basic labels and generic augmentation. We use the paper’s original GroundFscore (GF), with the scoring rule, audits, and expert references fixed across methods and within each testbed.
Table 6. Controlled real-robot study. Entries are FT / MDC / PerturBot. Values are percentages except GF.
| Family | Aligned success ↑ | Conflict success ↑ | Shortcut-following error ↓ | GF ↑ | |
|---|---|---|---|---|---|
| Vision | 0.50 | 84 / 85 / 86 | 83 / 84 / 85 | 7 / 6 / 5 | 0.66 / 0.68 / 0.70 |
| Vision | 0.75 | 87 / 88 / 88 | 70 / 75 / 84 | 21 / 16 / 6 | 0.54 / 0.60 / 0.69 |
| Vision | 0.95 | 89 / 89 / 90 | 49 / 60 / 82 | 43 / 31 / 8 | 0.36 / 0.47 / 0.67 |
| Language | 0.50 | 82 / 83 / 84 | 81 / 82 / 83 | 10 / 9 / 8 | 0.64 / 0.66 / 0.68 |
| Language | 0.75 | 86 / 87 / 88 | 67 / 73 / 82 | 24 / 18 / 9 | 0.50 / 0.57 / 0.67 |
| Language | 0.95 | 88 / 89 / 90 | 43 / 55 / 80 | 49 / 36 / 10 | 0.30 / 0.43 / 0.65 |
| Motor context | 0.50 | 80 / 81 / 82 | 79 / 80 / 81 | 12 / 11 / 10 | 0.60 / 0.62 / 0.65 |
| Motor context | 0.75 | 84 / 85 / 86 | 64 / 71 / 80 | 27 / 20 / 10 | 0.47 / 0.54 / 0.64 |
| Motor context | 0.95 | 86 / 87 / 88 | 39 / 52 / 77 | 53 / 39 / 12 | 0.27 / 0.40 / 0.62 |
The gains concentrate under conflict as training correlation strengthens. PerturBot’s conflict-success gains over FT grow from 2 points per family at to 33, 37, and 38 points at , while aligned-success gains at the latter setting remain only 1–2 points. PerturBot also exceeds MDC by 22–25 conflict-success points. Its GF advantage over FT grows from 0.04–0.05 to 0.31–0.35, alongside fewer shortcut-following errors.
These gains preserve responsiveness to task evidence. Paired tests assess whether decisions remain correct when the cue changes and switch correctly when the evidence changes. Both trials must be correct; inactivity does not count as invariance.
Table 7. Paired decision tests at , using the same trials as Table 6. Entries are FT / MDC / PerturBot (%).
| Family | Correct under both cue values ↑ | Correct switch with task evidence ↑ |
|---|---|---|
| Vision | 52 / 65 / 89 | 54 / 67 / 88 |
| Language | 47 / 60 / 87 | 49 / 63 / 86 |
| Motor context | 44 / 57 / 84 | 46 / 60 / 83 |
Together, these results support reduced reliance on the designed shortcuts, rather than general input insensitivity. Benefits are small at , though not demonstrably zero, and become substantially larger under stronger cue–evidence conflict.
Response to Reviewer RbAn (1/4)
We thank the reviewer for the detailed assessment and for recognizing the theoretical motivation, GroundFscore, and the real-robot factorial study. We address the concerns below.
We agree that aggregate success alone does not establish reduced shortcut reliance. We apologize for the oversight: the +12.2-point comparison used a published π_{0.5} baseline rather than a matched one. We therefore add an in-house FT baseline with the same initialization, task demonstrations, and training settings as PerturBot
Table 1. RoboTwin success (%). The published baseline is included for reference; our FT baseline uses the matched recipe.
Against the matched FT baseline, PerturBot improves Clean2Random-randomized success by 10.6 points; removing V reduces success by 8.4 points.
We also directly evaluate salience capture, noun lock-in, and motor inertia on RoboTwin.
Table 2. Direct shortcut evidence. Entries are matched FT / PerturBot. Vision and motor failure and success use separate probe sets; language outcomes come from the verb-change trials.
PerturBot reduces shortcut failures by 24, 33, and 22 points, respectively, while increasing task-relevant success.
We will also remove “the best randomized-scene result among the listed systems” and separately report the higher published results of LingBot-VA, ABot-M0.5, and GigaBrain-0.7. We do not claim state-of-the-art benchmark performance.