acceptodds
Under review as a conference paper at ICLR 2027

PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Abstract

A vision–language–action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RbAnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Summary

The paper argues that VLA policies often rely on modality shortcuts: cues that are reliable in successful demonstrations, such as wrist-view salience, the object noun, or the close-then-lift sequence, which replace the task evidence a decision should depend on. It contributes three things:

  • An information-theoretic analysis of when such shortcuts arise.
  • Perturbot, three training-data interventions:
    • V: wrist-view distractor insertion;
    • C: decision-relevant detailed captions;
    • R: random and failed trajectories relabeled with their local behavior.
  • GroundFscore (GF), an offline metric built from paired null and causal input edits.

With a π0.5 backbone, Perturbot raises real-robot General PnP success from 42% to 84% and improves RoboTwin 2.0 results.

Critical strengths

Strengths

  1. Clear theoretical motivation. Sec. 3.1 unifies three seemingly different failures (salience capture, noun lock-in, motor inertia) under one information-theoretic view. It explains why more demonstrations of the same kind need not remove them (collinearity and degeneracy), and it derives two distinct data-level remedies: lowering redundancy and raising evidence informativeness.

  2. GroundFscore is well designed.

    • Combining stability (1 − R_c) and causal sensitivity (S_c) in a harmonic mean rules out degenerate solutions. A policy that ignores its input scores zero, and so does one that reacts to everything.
    • Projecting the response onto the required change Δa* rewards only movement in the correct direction.
    • It needs only paired forward passes with shared sampling noise, so it is cheap and model-agnostic.
    • It predicts online failures much better than task success does (Spearman 0.72 vs. 0.31, Table 13).

    An offline diagnostic of this kind is a useful contribution in its own right.

  3. Thorough real-robot ablation. The full V/C/R factorial is run together with budget-matched mechanism controls.

Critical weaknesses

Weaknesses

W1. RoboTwin success gains do not show that shortcuts were reduced

The central claim is that Perturbot improves VLAs by reducing shortcuts. RoboTwin 2.0, however, is not designed to isolate shortcuts. Its failures can stem from manipulation precision, bimanual coordination, long horizons, or contact dynamics. A higher success rate therefore does not show that the improvement comes from reduced shortcut reliance rather than generic gains from extra data coverage (R) or richer supervision (C). GF, which the paper proposes for exactly this purpose, is not reported on RoboTwin.

Two issues further weaken the RoboTwin evidence:

  • No matched baseline. The π0.5 row in Table 2 is the published number from Cai et al. (2026b), identical to Table 7, whereas the Perturbot rows use the authors' own fine-tuning recipe. The headline +12.2-point gain thus compares two different training recipes.
  • Selective presentation. Main-text Table 2 lists only systems that Perturbot outperforms, and the text claims "the best randomized-scene result among the listed systems." Appendix Table 7 contains many stronger systems (see W2).

W2. Strong models outperform Perturbot without it; whether the method helps them is unknown

Several open-source models with released RoboTwin 2.0 checkpoints or post-training code outperform Perturbot by about 10 points without any shortcut-targeted augmentation:

  • Full: LingBot-VA 92.2 and ABot-M0.5 94.1, vs. 83.8 for Perturbot;
  • Clean2Random-Randomized: GigaBrain-0.7 67.9, vs. 58.2 for Perturbot. The paper itself calls this setting "the more direct test of modality shortcuts."

This suggests either that shortcuts are not the dominant failure source, or that strong models already have low shortcut reliance. In both cases the practical value of Perturbot is unclear. Since all experiments use only π0.5, I ask the authors to:

  1. report GF for one or two of these strong models, to show whether they suffer from shortcuts at all;
  2. apply Perturbot to at least one of them on RoboTwin.

If GF does not improve further, or success drops, the proposed augmentation scheme would be of limited use or would even conflict with what strong models already learn. The method's benefit may then be confined to compensating for a weaker baseline.

W3. A dedicated shortcut testbed is needed

To support the claim that the method works because it reduces shortcuts, the authors should evaluate on a setting designed to expose them, rather than relying on RoboTwin success rates. The existing probes (50-trial salience-capture and motor-inertia probes on the real robot, and 5 verb-change task pairs in simulation) are small. A controlled simulated testbed would be more convincing: vary the correlation between the shortcut cue and the task evidence in the training data, and test under cue–evidence conflict. If Perturbot targets shortcuts, its benefit, in both success and GF, should grow with the correlation strength and vanish without it.

W4. Limited methodological novelty

Each intervention closely resembles existing techniques:

  • V: generative or semantic image augmentation;
  • C: instruction relabeling or enrichment;
  • R: hindsight relabeling of play data (Lynch et al., 2020).

The contribution therefore lies mainly in their combination rather than in new methodology. The paper also misses the most closely related work on shortcut learning in generalist robot policies: Xing et al., "Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation" (arXiv 2508.06426). That work also attributes shortcuts to dataset structure and mitigates them with targeted data augmentation, and the paper should discuss how its approach differs.

Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Response to Reviewer RbAn (1/4)

We thank the reviewer for the detailed assessment and for recognizing the theoretical motivation, GroundFscore, and the real-robot factorial study. We address the concerns below.

💬 W1: RoboTwin success gains do not establish reduced shortcut reliance, and the baseline comparison needs clarification.

We agree that aggregate success alone does not establish reduced shortcut reliance. We apologize for the oversight: the +12.2-point comparison used a published π_{0.5} baseline rather than a matched one. We therefore add an in-house FT baseline with the same initialization, task demonstrations, and training settings as PerturBot

Table 1. RoboTwin success (%). The published baseline is included for reference; our FT baseline uses the matched recipe.

MethodFull: cleanFull: randomizedClean2Random: cleanClean2Random: randomized
π_{0.5}, published82.776.870.746.0
π_{0.5} FT, our recipe81.475.669.847.6
PerturBot w/o V84.979.372.649.8
PerturBot w/o C84.180.671.855.4
PerturBot w/o R84.480.972.056.2
PerturBot85.681.973.458.2

Against the matched FT baseline, PerturBot improves Clean2Random-randomized success by 10.6 points; removing V reduces success by 8.4 points.

We also directly evaluate salience capture, noun lock-in, and motor inertia on RoboTwin.

Table 2. Direct shortcut evidence. Entries are matched FT / PerturBot. Vision and motor failure and success use separate probe sets; language outcomes come from the verb-change trials.

FamilyGroundFscoreShortcut failure (%)Task-relevant success (%)
Vision: salience capture0.33 / 0.5146 / 2258 / 74
Language: noun lock-in0.27 / 0.5552 / 1936 / 71
Motor context: motor inertia0.30 / 0.4840 / 1854 / 70

PerturBot reduces shortcut failures by 24, 33, and 22 points, respectively, while increasing task-relevant success.

We will also remove “the best randomized-scene result among the listed systems” and separately report the higher published results of LingBot-VA, ABot-M0.5, and GigaBrain-0.7. We do not claim state-of-the-art benchmark performance.

AuthorsOpenReview ↗

Response to Reviewer RbAn (2/4)

💬 W2: Do stronger models still exhibit shortcuts, and can they benefit from PerturBot?

We agree that results on π0.5\pi_{0.5} alone do not establish the method's utility for stronger models. We therefore evaluate LingBot-VA [1] and ABot-M0.5 [2] under Full, and GigaBrain-0.7 [3] under Clean2Random (clean-only Co-Train). Within each backbone, we compare our reproduced RoboTwin baseline with the same training recipe augmented by PerturBot, matching initialization, base task demonstrations, and training budget.

Shortcut reliance and mitigation. All three reproduced baselines still exhibit salience capture, noun lock-in, and motor inertia. Adding PerturBot reduces every shortcut failure rate while improving GroundFscore and RoboTwin success (Table 3). Average success increases by 0.5, 0.5, and 1.3 points for LingBot-VA, ABot-M0.5, and GigaBrain-0.7.

Table 3. RoboTwin shortcut diagnostics. Each entry is Baseline → + PerturBot. All values are percentages except GF. SC/NL/MI denote salience capture, noun lock-in, and motor inertia; GF is the V/L/A mean. Failure definitions follow W1. Success is the average in Table 4. Comparisons are within each backbone.

Backbone (protocol)SC failure ↓NL failure ↓MI failure ↓Mean GF ↑RoboTwin success ↑
LingBot-VA (Full)16 → 1318 → 1214 → 110.58 → 0.6092.2 → 92.7
ABot-M0.5 (Full)17 → 1423 → 1615 → 120.53 → 0.5694.0 → 94.5
GigaBrain-0.7 (Clean2Random)21 → 1425 → 1619 → 120.48 → 0.5266.9 → 68.2

Nominal benchmark performance. Table 4 reports the published results, our reproduced baselines, and + PerturBot. Each backbone follows its corresponding training protocol, with clean, randomized, and average success reported separately. We compare PerturBot with our own baseline, rather than with published numbers or models trained under different protocols.

Table 4. RoboTwin success (%). Baseline denotes our reproduction of each model's RoboTwin training recipe. Reported results are from [1–3].

Backbone (protocol)Result / ConditionCleanRandomizedAverage
LingBot-VA (Full)Reported [1]92.9391.5592.24*
Baseline92.691.892.2
+ PerturBot93.292.292.7
ABot-M0.5 (Full)Reported [2]94.0094.2094.10
Baseline93.894.294.0
+ PerturBot94.494.694.5
GigaBrain-0.7 (Clean2Random)Reported [3]66.8067.9067.35
Baseline66.467.466.9
+ PerturBot67.369.168.2

LingBot-VA's 92.24 is computed from its reported Easy/Hard results. Bold highlights our method, not the best published score.

Relative to our reproduced baselines, PerturBot improves Full-average success by 0.5 points for both LingBot-VA and ABot-M0.5, and Clean2Random-randomized success by 1.7 points for GigaBrain-0.7. These nominal point gains are modest but accompany clearer improvements on the shortcut probes. Our claim concerns improved task-relevant evidence use, rather than a new benchmark ranking.

Real-robot validation. On General PnP, PerturBot raises task success by 16, 12, and 18 points and improves GroundFscore in every family and on average (Table 5).

Table 5. Real-robot General PnP. Entries are Baseline → + PerturBot. Task success is the success rate on the General PnP cases. GF$_V$, GF$_L$, and GF$_A$ are GroundFscore on held-out observations, and Mean GF is their average.

BackboneTask success ↑GF$_V$ ↑GF$_L$ ↑GF$_A$ ↑Mean GF ↑
LingBot-VA58 → 740.41 → 0.560.34 → 0.510.36 → 0.470.37 → 0.51
ABot-M0.564 → 760.44 → 0.570.39 → 0.530.38 → 0.480.40 → 0.53
GigaBrain-0.752 → 700.35 → 0.530.30 → 0.490.29 → 0.450.31 → 0.49

Together, these results support PerturBot as complementary to stronger pretraining, rather than merely compensating for a weaker backbone. They motivate scaling decision-relevant annotations and behavioral coverage alongside data quantity, instead of assuming that scale alone eliminates shortcuts.

References

[1] Lin Li et al. Causal World Modeling for Robot Control. arXiv preprint arXiv:2601.21998, 2026. https://arxiv.org/abs/2601.21998

[2] Ronghan Chen et al. ABot-M0.5: Unified Mobility-and-Manipulation World Action Model. arXiv preprint arXiv:2607.00678, 2026. https://arxiv.org/abs/2607.00678

[3] GigaBrain Team et al. GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture. arXiv preprint arXiv:2608.15875, 2026. https://arxiv.org/abs/2608.15875

AuthorsOpenReview ↗

Response to Reviewer RbAn (3/4)

💬 W3: A dedicated shortcut testbed is needed to establish the mechanism.

We directly vary training cue–decision agreement, q∈{0.50,0.75,0.95}q\in\{0.50,0.75,0.95\}, while keeping cue and decision frequencies balanced. We evaluate three controlled real-robot testbeds:

  • Vision: a task-irrelevant salient marker predicts the target’s side, while the instruction and object identity determine the correct target.
  • Language: object identity predicts a habitual pick or push operation, while the instruction’s verb determines the required operation. Both operations have valid demonstrations for every object.
  • Motor context: two observable post-grasp checkpoints, 8 cm apart with the same height and orientation, predict habitual transport or regrasp. The actual grasp outcome determines the correct continuation: transport when holding the object, and regrasp when empty. All four checkpoint–outcome combinations have valid expert continuations. We evaluate continuation choices, not identical continuous actions across checkpoints.

Each setting uses 400 base demonstrations, three training seeds, and 50 held-out layouts evaluated under all four cue–evidence combinations. Methods share initialization, base task data, and optimization budget. The matched-data control (MDC) also shares PerturBot’s auxiliary data but uses basic labels and generic augmentation. We use the paper’s original GroundFscore (GF), with the scoring rule, audits, and expert references fixed across methods and qq within each testbed.

Table 6. Controlled real-robot study. Entries are FT / MDC / PerturBot. Values are percentages except GF.

FamilyqqAligned success ↑Conflict success ↑Shortcut-following error ↓GF ↑
Vision0.5084 / 85 / 8683 / 84 / 857 / 6 / 50.66 / 0.68 / 0.70
Vision0.7587 / 88 / 8870 / 75 / 8421 / 16 / 60.54 / 0.60 / 0.69
Vision0.9589 / 89 / 9049 / 60 / 8243 / 31 / 80.36 / 0.47 / 0.67
Language0.5082 / 83 / 8481 / 82 / 8310 / 9 / 80.64 / 0.66 / 0.68
Language0.7586 / 87 / 8867 / 73 / 8224 / 18 / 90.50 / 0.57 / 0.67
Language0.9588 / 89 / 9043 / 55 / 8049 / 36 / 100.30 / 0.43 / 0.65
Motor context0.5080 / 81 / 8279 / 80 / 8112 / 11 / 100.60 / 0.62 / 0.65
Motor context0.7584 / 85 / 8664 / 71 / 8027 / 20 / 100.47 / 0.54 / 0.64
Motor context0.9586 / 87 / 8839 / 52 / 7753 / 39 / 120.27 / 0.40 / 0.62

The gains concentrate under conflict as training correlation strengthens. PerturBot’s conflict-success gains over FT grow from 2 points per family at q=0.50q=0.50 to 33, 37, and 38 points at q=0.95q=0.95, while aligned-success gains at the latter setting remain only 1–2 points. PerturBot also exceeds MDC by 22–25 conflict-success points. Its GF advantage over FT grows from 0.04–0.05 to 0.31–0.35, alongside fewer shortcut-following errors.

These gains preserve responsiveness to task evidence. Paired tests assess whether decisions remain correct when the cue changes and switch correctly when the evidence changes. Both trials must be correct; inactivity does not count as invariance.

Table 7. Paired decision tests at q=0.95q=0.95, using the same trials as Table 6. Entries are FT / MDC / PerturBot (%).

FamilyCorrect under both cue values ↑Correct switch with task evidence ↑
Vision52 / 65 / 8954 / 67 / 88
Language47 / 60 / 8749 / 63 / 86
Motor context44 / 57 / 8446 / 60 / 83

Together, these results support reduced reliance on the designed shortcuts, rather than general input insensitivity. Benefits are small at q=0.50q=0.50, though not demonstrably zero, and become substantially larger under stronger cue–evidence conflict.

Reviewer_PTt8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PerturBot addresses visual, language, and motor shortcuts in VLA policies through task-preserving wrist-image edits, behavior-grounded captions, and relabeled random or failed trajectory segments. It also proposes GroundFscore, combining invariance to irrelevant edits with responsiveness to relevant edits. Experiments with π0.5 cover real-robot pick-and-place, RoboTwin 2.0, factorial ablations, data scaling, and the relationship between the offline score and online failures.

Critical strengths
  1. The shortcut examples identify concrete failure modes that ordinary task success can miss. Pairing irrelevant and task-changing interventions is a useful way to distinguish robustness from simply ignoring a modality.

  2. The real-robot factorial study is strong. All eight component combinations and several budget-matched controls are evaluated, with success improving from 42% to 84% for the full method (Table 1).

  3. The paper checks controllability and reports uncertainty for the offline-to-online correlation. It also carefully limits the interpretation of relabeled failures: these expand behavioral support but do not demonstrate recovery under the original goal.

Critical weaknesses
  1. GroundFscore's causal term only measures a clipped projection onto the desired action change (Eq. 3). A response equal to the target change plus an arbitrarily large orthogonal error still scores 1; overshooting in the target direction is also unpenalized. This weakens its interpretation as correct evidence use, even as a diagnostic.

  2. The 'eight times fewer demonstrations' comparison retains all 400 auxiliary recordings at every PerturBot budget (Appendix B). Matching optimizer steps does not match total collection and annotation effort, so the result establishes task-demonstration efficiency rather than overall data efficiency.

  3. The metric is validated on 100 held-out pick-and-place observations and 36 related π0.5 checkpoints. This is useful initial evidence, but does not establish that it predicts failures for other skills or policy families, despite the broader scaling motivation.

Questions for authors
  1. How often do large orthogonal errors or overshooting occur in the causal probes? Do the reported rankings persist with a score that penalizes the full action-change error?

  2. How does the scaling comparison change when auxiliary collection time and editing/captioning effort are included in the data budget?

  3. Does GroundFscore predict held-out task or intervention families, for example the RoboTwin verb-change probes, without retuning the score?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PTt8 (1/2)

We thank the reviewer for the positive assessment and constructive feedback. We address the concerns below.

💬 W1 & Q1: How often do overshoot and orthogonal errors occur, and do the rankings persist when these errors are penalized?

We measure these errors on the same 36 checkpoints and 100 held-out observations. For the predicted change δ=π(e(o))-π(o) and expert change t=Δa⋆t=\Delta a^\star, we compute

\beta=\frac{\lVert\delta-\alpha t\rVert}{\lVert t\rVert}.$$ α>1 indicates overshoot, and β is the normalized orthogonal residual. The thresholds α>1.25 and β>0.5 are fixed without tuning to online outcomes. *Table 1. Frequency and magnitude of causal-response errors. * | Family | Valid pairs | Overshoot: α>1 (%) | Substantial overshoot: α>1.25 (%) | Large orthogonal error: β>0.5 (%) | β: median / P90 | |:---:|:---:|:---:|:---:|:---:|:---:| | Vision | 91 | 16 | 3 | 12 | 0.18 / 0.53 | | Language | 88 | 13 | 1 | 17 | 0.33 / 0.58 | | Motor context | 85 | 18 | 4 | 9 | 0.17 / 0.51 | Overshoot is most frequent for motor context (18%), and large orthogonal error is most frequent for language (17%). Substantial overshoot (α>1.25) is uncommon, at 1–4%. We next test whether penalizing these errors changes the metric’s predictive value. GF-Full replaces only the causal term: $$S_c^{\mathrm{full}}=\mathrm{clip}_{[0,1]}\left(1-\mathbb{E}\left[\frac{\lVert\delta-\Delta a^{\star}\rVert}{\lVert\Delta a^{\star}\rVert}\right]\right).$$ The expectation is over valid causal edits of family c. Stability, harmonic aggregation, normalization, references, filtering, and paired decoding noise are unchanged. *Table 2. Spearman correlation with the negative online shortcut-failure rate on the same 36 checkpoints. Intervals are paired grouped bootstrap 95% CIs.* | | Predictor | Spearman [95% CI] | |:---:|:---:|:---:| | | Task success | 0.31 [−0.06, 0.60] | | | Original GroundFscore | 0.72 [0.49, 0.86] | | | **GF-Full** | **0.75 [0.53, 0.88]** | GF-Full retains the correlation advantage over task success (paired difference 0.44 [0.15, 0.69]) and preserves checkpoint rankings (Kendall’s τ_{b}=0.97). *Table 3. Condition-level scores and rankings. Rank changes are shown as Original GF → GF-Full.* | Condition | Original GF | GF-Full | Rank | Failure (%) ↓ | |:---:|:---:|:---:|:---:|:---:| | FT | 0.237 | 0.198 | 11 → 11 | 75 | | + V | 0.327 | 0.286 | 8 → 8 | 58 | | + C | 0.330 | 0.291 | 7 → 7 | 70 | | + R | 0.333 | 0.298 | 6 → 6 | 54 | | + V + C | 0.430 | 0.386 | **3 → 4** | 53 | | + V + R | 0.430 | 0.392 | 3 → 3 | 37 | | + C + R | 0.433 | 0.396 | 2 → 2 | 48 | | **Full PerturBot** | **0.573** | **0.528** | **1 → 1** | **22** | | Noise / dropout | 0.260 | 0.224 | 9 → 9 | 71 | | Twice the task demonstrations | 0.247 | 0.211 | 10 → 10 | 73 | | Negative guidance | 0.350 | 0.312 | 5 → 5 | 55 | | Wrist camera removed | 0.213 | 0.171 | 12 → 12 | 50 | Full PerturBot remains rank 1. The only change resolves the +V+C and +V+R tie in favor of +V+R. > 💬 **W2 & Q2**: *How does the scaling comparison change when auxiliary collection and editing/captioning effort are included?* The **eightfold reduction refers to task demonstrations, not total human effort**. The same scaling runs are restated below in person-hours. Every PerturBot scale uses the same 200 random and 200 failed trajectories, 1,400 retained segments, and sampling rates $\rho=0.30$, $p_V=p_C=0.50$. Auxiliary examples replace batch items and add no optimizer updates. Each standalone run is charged the full auxiliary preparation cost. - **Task collection:** 6 min per demonstration for both methods, including resets and quality control. - **Auxiliary collection:** 400 recordings × 4 min = 26.7 person-hours, including preparation, resets, and quality control. Clips last 5–10 s. - **Auxiliary review:** 1,400 segments × 1.5 min = 35.0 person-hours for segmentation, labels, and draft captions. - **Wrist-edit review:** one 1 min check for half of the unique demonstrations and auxiliary segments. The wrist-edit count is an offline preparation assumption, not a consequence of the training-time probability $p_V=0.50$. Totals are rounded only after summing the unrounded components. *Table 4. Estimated human effort. Rows are matched on demonstration count, not total effort. Success rates are point estimates. Robot occupancy and automated processing are excluded.* | Task demonstrations | FT effort (person-h) | PerturBot effort (person-h) | FT success (%) | PerturBot success (%) | |:---:|:---:|:---:|:---:|:---:| | 125 | 12.5 | 86.9 | 30 | 48 | | 250 | 25.0 | 100.4 | 36 | 66 | | 500 | 50.0 | 127.5 | 42 | 84 | | 1,000 | 100.0 | 181.7 | 48 | 96 | PerturBot reaches the same 48% success in 86.9 person-hours, against 100.0 for standard fine-tuning: a 13.1% reduction in estimated human effort, with an eightfold reduction in task demonstrations.
AuthorsOpenReview ↗

Response to Reviewer PTt8 (2/2)

💬 W3 & Q3: Does GroundFscore predict held-out probes, such as RoboTwin verb changes, without score retuning?

We evaluate both GF variants on five RoboTwin verb-change pairs excluded from score selection, with all scoring rules frozen. The original operations are included in policy training. Null edits are same-operation paraphrases; causal edits change the verb and use same-state expert continuations. Offline and online evaluations use disjoint scene seeds.

Table 5. Frozen language-family scores and online outcomes, averaged over the same three checkpoints per method, with 100 verb-change trials per checkpoint

MethodOriginal GF$_L$GF-Full$_L$Followed (%)Lock-in (%)Other (%)
FT0.270.24365212
PerturBot w/o V0.440.39642610
PerturBot w/o C0.340.29454312
PerturBot w/o R0.500.43662410
PerturBot0.550.49711910

PerturBot raises verb-following success from 36% to 71% and reduces lock-in from 52% to 19%, alongside higher scores under both GF variants.

We then measure predictive correlations across 15 individual checkpoints, not the five method means. Nominal task success measures performance under ordinary instructions rather than verb changes.

Table 6. Checkpoint-level Spearman correlations. The 95% CIs use paired bootstrap resampling over five training recipes and their checkpoints, not task pairs or trials

PredictorSpearman with negative lock-in [95% CI]Spearman with followed rate [95% CI]
Nominal task success0.14 [−0.72, 0.76]0.44 [−0.39, 1.00]
Original GF$_L$0.91 [0.54, 0.99]0.84 [0.29, 0.96]
GF-Full$_L$0.82 [0.27, 0.95]0.92 [0.52, 1.00]

Both frozen GF variants show higher point correlations than nominal task success, supporting transfer to these probes within the π0.5π_{0.5} family.

Thank you for the GF-Full analysis, cost breakdown, and held-out verb-change evaluation. These address most of my concerns and strengthen the evidence within pi0.5. Please keep the distinction between eightfold fewer task demonstrations and the estimated 13.1% reduction in human effort explicit. Generalization across policy families remains an open question, but my overall assessment stays positive. I will maintain my rating of 3 (weak acceptance).

Reviewer_dpkxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies modality shortcuts in vision-language-action policies. The proposed method, Perturbot, combines three training-time data interventions: task-preserving wrist-view perturbations, behavior-supported caption enrichment, and relabeled segments from random-motion and failed-execution recordings. The paper also proposes GroundFscore, an offline diagnostic combining stability under task-irrelevant edits with responsiveness to task-relevant edits. Experiments excuted on real-robot General Pick-and-Place and the RoboTwin 2.0 benchmark.

Critical strengths
  • Strong experimental decomposition. The complete three-component factorial study and budget-matched controls support the effectiveness of the data interventions.

  • Informative targeted evaluation. The distractor, grasp-outcome, and verb-change probes examine whether improvements preserve useful conditioning. These tests strengthen the interpretation that Perturbot reduces shortcut dependence.

Critical weaknesses
  • GroundFscore has a response-scoring blind spot. Its projection-based causal term can give full credit to overshooting responses or responses containing large orthogonal errors. The reported correlation is encouraging, but stronger validation is needed to establish whether this limitation affects model rankings.

  • Comparison and efficiency claims need clarification. The eightfold reduction in task demonstrations does not account for auxiliary-data collection and annotation costs.

Questions for authors
  • Does GroundFscore retain its correlation advantage when the causal term penalizes overshoot and orthogonal response errors?

  • Please also specify the auxiliary-data amounts and sampling rates used at each scaling budget.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dpkx

We thank the reviewer for the positive assessment. We address the concerns below.

💬 W1 & Q1: Does GroundFscore retain its correlation advantage when overshoot and orthogonal response errors are penalized?

We thank the reviewer for identifying this limitation. The original causal term measures directional responsiveness and can over-credit overshoot and orthogonal deviations. We recompute GF-Full on the same 36 checkpoints, 100 observations, expert references, and online outcomes, replacing only the causal term:

Scfull=clip[0,1](1−Eo,e∼Eccausal[∥δ−Δa⋆∥∥Δa⋆∥]),δ=π(e(o))−π(o).S_c^{\mathrm{full}}=\mathrm{clip}_{[0,1]}\left(1-\mathbb{E}_{o,e\sim\mathcal{E}^{\mathrm{causal}}_c}\left[\frac{\lVert\delta-\Delta a^{\star}\rVert}{\lVert\Delta a^{\star}\rVert}\right]\right),\quad\delta=\pi(e(o))-\pi(o).

Δa⋆\Delta a^{\star} is the expert action change.

In Table 1 we correlate task success, Original GF, and GF-Full with the negative online shortcut-failure rate, and in Table 2 we compare the twelve condition means and ranks.

Table 1. Spearman correlation with the negative online shortcut-failure rate.

PredictorSpearman [95% CI]
Task success0.31 [−0.06, 0.60]
Original GroundFscore0.72 [0.49, 0.86]
GF-Full0.75 [0.53, 0.88]

GF-Full retains a stronger correlation with the negative online shortcut-failure rate than task success. Its ranking remains highly consistent with Original GF (Kendall’s τ_b = 0.97).

Table 2. Condition means.

ConditionOriginal GFGF-FullRankFailure (%) ↓
FT0.2370.19811 → 1175
+ V0.3270.2868 → 858
+ C0.3300.2917 → 770
+ R0.3330.2986 → 654
+ V + C0.4300.3863 → 453
+ V + R0.4300.3923 → 337
+ C + R0.4330.3962 → 248
Full PerturBot0.5730.5281 → 122
Noise / dropout0.2600.2249 → 971
Twice the task demonstrations0.2470.21110 → 1073
Negative guidance0.3500.3125 → 555
Wrist camera removed0.2130.17112 → 1250

PerturBot remains rank 1, above FT. The only rank change is +V+R above +V+C. Penalizing overshoot and orthogonal error leaves the correlation advantage intact.

💬 W2 & Q2: Auxiliary-data quantities, sampling rates, and the cost of the scaling comparison

The eightfold reduction is 125 versus 1,000 task demonstrations. It counts task demonstrations, and the collection and annotation time is estimated below.

ρ is the probability of sampling an auxiliary example. p_V and p_C apply to both task demonstrations and auxiliary examples, which replace part of each batch and do not add optimizer updates. Every scale uses the same auxiliary pool and the same sampling rates.

Table 3. Auxiliary-data quantities and sampling rates. Trajectory and segment counts are reported as random / failed.

Task demonstrationsAuxiliary trajectoriesRetained segmentsρp_Vp_C
125200 / 200760 / 6400.300.500.50
250200 / 200760 / 6400.300.500.50
500200 / 200760 / 6400.300.500.50
1,000200 / 200760 / 6400.300.500.50

The 200 / 200 pool is a controlled choice, not a minimum requirement. Table 4 ablates whether the 125-demonstration run needs that full pool.

Table 4. Auxiliary-data budget at 125 demonstrations

MethodRandom trajectoriesFailed trajectoriesTask success (%) ↑
Standard FT0030
FT + PerturBot252540
FT + PerturBot505043
FT + PerturBot10010047
FT + PerturBot — main setting20020048

PerturBot already performs well with very few auxiliary trajectories. In the main scaling experiment we still use the full pool, so that every scale shares the same auxiliary setting.

In Table 5 we estimate the human effort of the original eightfold comparison.The 12.5 h is 125 task demonstrations at 6 min each. The 74.4 h is 26.7 h for 400 auxiliary recordings at 4 min each, including preparation, resets, and quality control (clips last 5–10 s), 35.0 h for one 1.5 min review of each of the 1,400 segments, and 12.7 h for 1 min wrist-edit checks on half of the demonstrations and segments at p_V = 0.50.

Table 5. Estimated human effort. Auxiliary counts are random / failed.

RecipeTask demonstrationsAuxiliary trajectoriesDemo collection (h)Auxiliary collection + review (h)Total human effort (h)Success (%)
Standard FT1,0000 / 0100010048
FT + PerturBot125200 / 20012.574.486.948

PerturBot also costs less: it reaches the same 48% success in 86.9 h, against 100 h for standard fine-tuning.

We have also updated the paper PDF with these additional experimental results.

Thank you for the detailed response and additional experiments. The new results and clarifications have addressed most of my concerns, particularly regarding the robustness of GroundFscore and the auxiliary-data costs. These additions provide more support for the paper's conclusions.

We sincerely thank the reviewer for their time and effort and will further improve the paper in the camera-ready version.

Comments on the paper

1 comment

Dear Reviewers, With the rebuttal period ending on November 18, please take a moment to check the authors' latest responses, post your feedback, and join the discussion. We greatly appreciate your hard work and dedication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.