acceptodds
Under review as a conference paper at ICLR 2027

What Success Rate Hides: Mapping Disturbance Tolerance in Vision-Language-Action Policies

Abstract

Vision-language-action (VLA) policies are typically ranked by task success, but success captures only the endpoint of behavior rather than how a policy responds during execution. Two policies that both complete a task from a nominal state may behave very differently when the same in-progress state is physically displaced. We measure this directly using a calibrated, policy-free push that produces a single mid-task release state, which is restored bit-identically for every policy. A disturbance-tolerance map then records next-milestone success across push directions and severities, while a scalar score, Qµ, summarizes performance under a declared disturbance distribution and the full map reveals where policy behaviors diverge. Across three released LIBERO policies, clean success does not resolve disturbance tolerance: 18 of 20 fresh anchors at which all policies are clean-competent separate under at least one matched disturbance, and all 28 selected decisive cells preserve their performance gap after expansion to 10 stochastic draws, including local reversals of the aggregate ranking. These differences persist even when execution cadence is equalized. VLA evaluation should therefore report clean competence, the tolerance score, and the disturbance-tolerance map, together with the execution setting under which they were obtained.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning, policy evaluation, disturbance tolerance, off-nominal behavior, optimal experimental design, embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VoKgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates how VLA policies recover from controlled disturbances during manipulation. It proposes disturbance-tolerance maps to reveal behavioral differences that ordinary task-success rates can hide.

Critical strengths
  1. Evaluating recovery and robustness beyond nominal task success is an essential component on understanding robot policy performances.

  2. The experiments compare three policies under matched disturbances and include repeated trials and controls for execution cadence.

Critical weaknesses
  1. The writing and organization make the experimental setup difficult to follow. The relationships between local-response probes, tolerance maps, and aggregate scores are not explained clearly enough.
  2. The manuscript introduces several experimental panels without a concise overview of their purposes and connections. Essential protocol details are scattered across the appendices.
  3. The theoretical propositions establish general possibilities, but their practical contribution to the proposed evaluation is insufficiently demonstrated.
  4. The central evidence is limited to three policies in LIBERO simulation. The headline result of differences at 18/20 anchors is conditional on a small subset where all policies pass the clean-success screen, which limits its generality.
  5. The introduction contains no citations to support its motivation or position the contribution relative to prior work on robustness and recovery evaluation. This makes the research gap and novelty difficult to assess.
  6. How can a ICLR submission cite other ICLR submissions? Also, both citations seem not to be relevant with the theme of the paper.
Questions for authors

The paper needs substantial reorganization and clearer writing. Present the complete evaluation protocol first, explain each experimental panel in one place, and distinguish the main tolerance-map contribution from the supporting local-response analysis. State the conclusions more narrowly around the tested policies and disturbances.

6 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have clarified the evaluation protocol, its theoretical motivation, experimental scope, and relationship to prior work, as detailed below.

[W1 / W2 / Q] A Complete Protocol and Unified Experimental Roadmap. The writing and organization make the experimental setup difficult to follow. The relationships between local-response probes, tolerance maps, and aggregate scores are not explained clearly enough.

The manuscript introduces several experimental panels without a concise overview of their purposes and connections. Essential protocol details are scattered across the appendices.

The paper needs substantial reorganization and clearer writing.

Present the complete evaluation protocol first, explain each experimental panel in one place, and distinguish the main tolerance-map contribution from the supporting local-response analysis.

We reorganized Method §3 to explain the complete measurement in one sequence, illustrated by Figure 2: (1) choose a saved pick/transport state jointly passed in undisturbed screening; (2) create a policy-free, 0.4-s end-effector translation along six task-frame directions and magnitudes 16/32/48 mm; (3) save the resulting release and restore it identically for each policy; (4) resume each policy with an empty action queue and a fresh query; (5) report the condition-wise recovery profile, the policy-pair difference, and an aggregate score under declared evaluation weights.

A recovery profile maps each physical condition to the probability of reaching the next milestone within 6 s; a contrast subtracts the profiles of two policies starting from the same release; an aggregate score averages one profile under an explicitly chosen condition distribution. The separate local-response probe and its propositions have been removed from the main argument.

Table R6. Research questions, main findings, and corresponding results.

QuestionEvidence and key resultCurrent main location
Are large gaps larger than rollout noise, and do they recur?Original 18/20 vs. 6.7 expected; additional-state comparisons; selected-condition confirmation 28/28§5.1, Fig. 3, pp. 6–7
Can the better policy change with the disturbance?At one state, near-equal scores 0.761 / 0.750 yet opposite confirmed winners; four further selected reversalsFig. 1, p. 2; §5.2, Fig. 4, pp. 7–8
What predictive information is lost in an aggregate?Same-release test Brier loss 0.0235 (profile) vs. 0.1094 (scalar)§5.3, Table 2, pp. 8–9
What does the physical program itself contribute?On 25 tasks, 48 mm vs. zero command: SmolVLA −0.34, p = 0.003; GR00T −0.11, p = 0.13§5.4, Fig. 5, pp. 9–10

All evaluation sets and rollout counts are described together in §4 / Table 1 (p. 5) rather than dispersed across result sections. Appendices A and C give reproducible protocol and study details; Appendices D–F follow the main-results order. This layout separates primary evidence, selected-condition confirmation, retrospective prediction, and exploratory controls. Three short supplementary replay videos illustrate map construction, policy differences and the six-direction map; they are visualizations of logged simulation actions, not new experimental samples.

We have reorganized the protocol in §3 / Figure 2 (pp. 3–5), consolidated the experiments in §4 / Table 1 (p. 5), and organized the results around Figures 1 and 3–5 (pp. 2, 6–9) and Table 2 (p. 8). Technical and experimental details are in Appendices A (pp. 14–22), C (pp. 23–30), and D–F (pp. 30–41).

[W3] The Role and Practical Meaning of the Remaining Lemma. The theoretical propositions establish general possibilities, but their practical contribution to the proposed evaluation is insufficiently demonstrated.

We removed the local-response propositions, which were not necessary to the central measurement result. The revised paper retains one simple dominance lemma directly linked to the score–profile relationship. Specifically, if policy A recovers at least as often as policy B in every shared condition, A's aggregate score is at least as high under any weighting over those conditions. Conversely, if A leads in some conditions and B leads in others, a ranking by one weighted average is not universal.

This is the reason we report the complete recovery profile and pairwise contrasts alongside the aggregate: they show which release conditions drive a score and whether the ordering depends on the evaluator's weighting. The lemma is not presented as a novel general theory.

At the book-to-caddy state, the original five-rollout maps yield aggregate scores 0.761 (SmolVLA) and 0.750 (GR00T), but new rollouts from two separate 48-mm releases give 20/20 vs. 0/20 and 8/20 vs. 20/20. Both policies therefore lead in a measured condition. The empirical reversal makes clear why the aggregate ranking is not invariant to the conditions emphasized.

We present the score–profile lemma in §3.2 (pp. 4–5), its concrete interpretation in §5.2 (pp. 7–8; Figure 1, p. 2), and its proof and fresh-rollout illustration in Appendices A.3 (p. 22) and D.1 (pp. 30–31).

[W4 / Q] Screened Subset, Policy Breadth, and Correct Inference Scope. The central evidence is limited to three policies in LIBERO simulation. The headline result of differences at 18/20 anchors is conditional on a small subset where all policies pass the clean-success screen, which limits its generality.

State the conclusions more narrowly around the tested policies and disturbances.

We have clarified that the headline 18/20 result is conditional on a joint undisturbed screen. It concerns 20 admitted states from 7 tasks, not 20 independent tasks, and tests the SmolVLA–GR00T pair specifically. Its pair-specific rollout-noise expectation is 6.7 (Bonferroni p = 1.02×10⁻⁹), with all 7/7 task roots above their null (p = 0.003). The earlier any-pair 18/20 interpretation was corrected: its null is 17.4 and p = 0.428, so it is not evidential.

To assess stability, we re-screened the 20 original states using 40 new undisturbed rollouts per stochastic policy (1,595/1,600 successes at both endpoints). Among 18 states passing 40/40 for both policies, 16 retain an original gap ≥0.6 (null 6.4, p = 4.5×10⁻⁸). The conservative clean-failure test remains non-rejecting (Berger–Boos p = 0.12), so we do not infer identical underlying clean-success probabilities.

The breadth of evidence is now greater, but comparisons retain their design boundaries:

Table R7. Additional evidence and the limits of the tested comparisons.

Additional evidenceResult
62 nonoverlapping new demonstration states from 33 existing tasksOFT–GR00T 21/30; π0.5–GR00T 15/29 in set A and 17/32 in set B (separate comparisons)
Independent confirmation at selected releases28/28 6-s gaps reproduced on 20 new rollouts per stochastic policy; 21/28 at 11 s
Model coverageFour released policies in total, but no state shared by all four and no direct SmolVLA–π0.5 pair
ScopeLIBERO simulation, saved pick and transport states; not a real-robot or dexterous-manipulation result

Thus, the supported conclusion is that large condition-wise recovery differences are measurable and reproducible within these tested, jointly screened comparisons. We do not claim a four-policy ranking, gap prevalence across arbitrary new tasks, or direct deployment benefit.

We have clarified the scope and inference units in §4 / Table 1 (p. 5), §5.1 / Figure 3 (pp. 6–7), Discussion §6 (p. 10), and Appendices C.1–C.5 (pp. 23–30).

Reviewer_az9COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares VLA policies from identical disturbed release states using direction–severity maps and an aggregate score. Experiments with three LIBERO policies reveal recovery differences hidden by similar clean success, including differences that persist at a common execution cadence.

Critical strengths
  1. The matched release-state protocol is useful and avoids confounding policy comparisons with different physical disturbances.
  2. Fresh candidate anchors and additional draws strengthen the empirical evidence.
  3. Common-cadence experiments and query-cost reporting clarify the effects of deployment settings.
Critical weaknesses
  1. Clean-competence screening remains uncertain. Appendix C requires five successful draws for stochastic policies; Section 4 retains 20 of 31 candidates. This does not establish exactly equal underlying clean-success probabilities. The 18/20 finding should therefore be interpreted conditionally on this screen. Stronger screening and results on excluded candidates would help assess its stability.
  2. Independent confirmation needs clarification. Appendix F states that selected cells were “expanded to 10 draws.” If discovery draws are included, the final gaps are not entirely independent replications. Since multiple directions and severities were searched per anchor, please report the new draws separately and specify the cell-selection rule.
  3. Local recovery may not reflect full-task robustness. Appendix C defines milestone success within six seconds and full-task success within eleven seconds, while the main results use the former. It also clears the action queue and re-queries the policy at release. Please validate whether this recovery measure after restarting execution reflects eventual task completion.
Questions for authors
  1. Does the 18/20 finding persist with stricter clean screening?
  2. Do the new draws alone reproduce the selected differences?
  3. Do milestone-tolerance maps agree with full-task success?
5 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address the questions regarding clean screening, independent confirmation, and evaluation endpoints below.

[W1 / Q1] Stability Under Stricter Undisturbed Screening. Clean-competence screening remains uncertain. Appendix C requires five successful draws for stochastic policies; Section 4 retains 20 of 31 candidates. This does not establish exactly equal underlying clean-success probabilities. The 18/20 finding should therefore be interpreted conditionally on this screen. Stronger screening and results on excluded candidates would help assess its stability.

Does the 18/20 finding persist with stricter clean screening?

We re-screened all 20 originally admitted states using 40 new undisturbed rollouts per stochastic policy, from the same saved zero-command releases as in the original screen. This was performed separately from disturbed evaluation.

Table R3. Original screening and stricter undisturbed re-screening.

CheckResult
Original eligibility20/31 candidate states from 7 tasks passed the undisturbed screen for all three original policies
Re-screened undisturbed success1,595/1,600 rollouts passed both the 6-s milestone and 11-s full-task endpoints
Strict eligibilityBoth SmolVLA and GR00T passed 40/40 at 18 states
Original disturbed large gaps in this strict subset16/18 states, versus 6.4 expected from rollout noise; Bonferroni p = 4.5×10⁻⁸
Re-check of originally excluded units11 excluded policy–state units were evaluated on 20 new rollouts each; none passed the undisturbed screen

The original finding is 18/20 SmolVLA–GR00T states with a condition gap ≥0.6 (null 6.7; Bonferroni p = 1.02×10⁻⁹), not a significant any-pair result: the any-pair count is 18/20 against 17.4 expected (p = 0.428). At the task-root level, all 7/7 original tasks exceed the pair-specific null (p = 0.003).

We also checked the more conservative question of whether uncertain clean-success probabilities alone could produce the observed gap count. The Berger–Boos test does not reject (p = 0.12). Thus, the strengthened screen supports the stability of the observed conditional comparison, but does not prove identical underlying clean competence or rule out every clean-failure explanation. The 16/18 calculation reuses the original disturbed outcomes and is explicitly labeled a retrospective sensitivity analysis, not an independent disturbed replication.

We present the stronger clean-screening analysis in §3.1 (pp. 3–4), §5.1 / Figure 3a (p. 6), and Appendix C.2 / Tables C.2–C.3 (pp. 25–27).

[W2 / Q2] New-Rollout-Only Confirmation and Condition Selection. Independent confirmation needs clarification. Appendix F states that selected cells were “expanded to 10 draws.” If discovery draws are included, the final gaps are not entirely independent replications. Since multiple directions and severities were searched per anchor, please report the new draws separately and specify the cell-selection rule.

Do the new draws alone reproduce the selected differences?

We conducted a confirmation analysis using only new rollouts, excluding all discovery draws and first re-runs.

Before collecting confirmation outcomes, a single largest-discovery-gap condition was chosen for each of the 18 large-gap states of the original panel and 10 earlier-panel states. The deterministic rule took the largest gap, then the higher magnitude, then a fixed direction order, and recorded the compared policy pair before any new outcome was collected. Every release was restored identically for the compared policies.

Table R4. Independent confirmation using new rollouts at 28 selected releases.

Independent confirmation, 28 previously selected conditionsResult
New measurements20 new rollouts per stochastic policy and condition (separate seeds)
Primary 6-s gap greater than 0.5 in its original direction28/28 conditions; minimum new gap 0.8, median 1.0
Multiplicity controlLargest Holm-adjusted p = 0.016 across all 28 conditions
Paired-seed sensitivity analysis27/28 conditions confirmed
Comparator type24 conditions compare a stochastic policy against deterministic OpenVLA-OFT; 4 compare two stochastic policies
Secondary 11-s criterion21/28 conditions met the criterion; none reversed sign

OpenVLA-OFT contributes replayed deterministic outcomes, not 20 independent stochastic samples. The inference for those 24 comparisons comes from the newly sampled stochastic policy. Because conditions were selected based on their discovery outcomes, 28/28 establishes repeatability of the selected differences at those releases, not their frequency among arbitrary tasks or states.

We describe condition selection and the new-rollout-only analysis in §4 / Figure 3b (p. 6), §5.1 (p. 7), and Appendices C.1 (pp. 24–25), C.4 / Table C.5 (pp. 27–29), and C.5 (pp. 29–30).

[W3 / Q3] Six-Second Recovery versus Eleven-Second Task Completion. Local recovery may not reflect full-task robustness. Appendix C defines milestone success within six seconds and full-task success within eleven seconds, while the main results use the former. It also clears the action queue and re-queries the policy at release. Please validate whether this recovery measure after restarting execution reflects eventual task completion.

Do milestone-tolerance maps agree with full-task success?

We scored both endpoints on the same rollouts: (i) reaching the next pick/transport milestone within 6 s, our primary recovery measurement, and (ii) completing the full LIBERO task within 11 s, the secondary endpoint. We also clarified exactly how execution restarts at a saved release.

Table R5. Six-second milestone recovery versus eleven-second task completion.

EvidenceSix-second milestoneEleven-second full task
Original panel, SmolVLA–GR00T large-gap states18/20The same 18/20; expected 8.0 under its noise null, p = 1.1×10⁻⁸
Previously selected conditions confirmed with 20 new draws28/28 gap confirmations21/28 satisfy the secondary criterion; no selected comparison reverses sign
Book-to-caddy reversal, two 48-mm releasesSmolVLA/GR00T 20/20 vs. 0/20 and 8/20 vs. 20/20The same success counts at both endpoints

Across original-panel disturbed rollouts, agreement between the two binary endpoints is 95.8%–97.5% for stochastic policies and 97.2% for deterministic OpenVLA-OFT. Among 147 conditions with large 6-s gaps, 137 keep their direction at 11 s. The endpoints are strongly related but not equivalent: for example, the new-start OpenVLA-OFT–GR00T study is significant at 6 s (21/30, p = 0.002) but not at 11 s (18/30, p = 0.093).

At each matched release, we clear the old action queue and query the policy afresh. We therefore test recovery after controlled restart from the same saved physical state, not uninterrupted execution along each policy’s own trajectory. Every policy retains its native query cadence.

We report both endpoints and the restart protocol in §3.1 (pp. 3–4), §5.1 / Figure 3 (pp. 6–7), and §5.2 (pp. 7–8; Figure 1, p. 2). Detailed endpoint comparisons are in Appendices C.3–C.5 / Table C.6 (pp. 27–30) and D.1 (pp. 30–31).

Reviewer_2BfYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on one of the central problems of VLA evaluation, i.e., is success-score a sufficient metric for policy evaluation. The paper argues that it is not and instead proposes that many papers that claim strong VLAs perform poorly (or their performance drops quite a bit) under disturbances. The paper formalizes this by assuming some anchor states (either while reaching an object or carrying it so just 2 modes) and then performing local perturbations at these anchor states and checking if the policy can reach the next intermediate goal within 6 seconds. The results look at 18 anchors over 8 LIBERO tasks and try to provide an exhaustive map of perturbations (ranging from 8-48 mm at fixed steps) and 6 directions. Comparing success rates from these 3x6 combinations with 5 different rollouts the paper finds that SmolVLA can have a large performance decrease while Groot and Open-VLA perform much better. However, the paper finds that the average score can mask local details. Specifically, they found some states where on SmolVLA succeeded while both Groot and Open-VLA failed.

Critical strengths

The paper focuses on an important problem of trying to better understand the robustness of VLA models.

Critical weaknesses

The paper is a bit hard to follow. As I mention below some mathematical symbols are introduced but not defined precisely. The overall idea is also not very novel in general. Specifically, taking some anchor states and perturbing them to check if goal is reached is something that we have been doing forever. I think the main contribution of the paper is just to formalize this approach a bit more but its unclear if the formulation is indeed clear.

Overall, since all the experiments are in simulation. Why cannot we do more rollouts from a larger pool of start state distributions even for subtasks. The local perturbations definitely are helpful but does a structured perturbation (e.g. having a vector in the gravity plane) really matter that much. Couldn’t some of the policies be biased towards these fixed states? It would be interesting to compare how this approach compares against running a lot more rollouts from intermediate states without this structured perturbation. Another, baseline that we often use is is to inject arbitrary noise in the actions (further away from goal) and decay that noise. How does that compare? Specifically, I think it’s quite well known that VLAs have high variance especially some of the academic ones. But while this paper focuses on an important problem, I think the lack of comparison on why their perturbation approach is better is a bit unclear to me. I guess another problem would be perturbations in joint space vs end-effector space. The paper only focuses on end-effector perturbations, but such perturbations are clearly biased since some joints may not change at all. Hence, joint space perturbations can be more information and also more challenging to do.

The kind of tasks that are being considered are a bit limited. The anchor is either the robot approaching an object or transporting it. While such pick-place can be a lot of manipulation VLAs shine a lot more on dextrous tasks which are hard to isolate in such states.

In line 96 the paper defines or uses “u” but never ever defines it mathematically. It’s suppsoed to be a small probe command but is it a one step or multi-step command. Are these perturbations completely in the EE space or the joint space. Its completely ungrounded. Same with the use of h which is referred to as h controls (the local response) but what is it really.

Questions for authors

see above

5 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address the concerns regarding the contribution, evaluation breadth, perturbation design, and task scope below.

[W1 / W9] Methodological Contribution, Shared Releases, and Mathematical Clarity. The paper is a bit hard to follow. As I mention below some mathematical symbols are introduced but not defined precisely. The overall idea is also not very novel in general. Specifically, taking some anchor states and perturbing them to check if goal is reached is something that we have been doing forever. I think the main contribution of the paper is just to formalize this approach a bit more but its unclear if the formulation is indeed clear.

In line 96 the paper defines or uses “u” but never ever defines it mathematically. It’s suppsoed to be a small probe command but is it a one step or multi-step command. Are these perturbations completely in the EE space or the joint space. Its completely ungrounded. Same with the use of h which is referred to as h controls (the local response) but what is it really.

We have clarified the contribution relative to existing approaches. Perturbing intermediate robot states is not new, and RoboRecover also compares policies from a shared deviation state; SMaRP does not claim to have introduced shared-state comparisons. The contribution is a reproducible, condition-resolved comparison protocol: (1) admit a saved mid-task state only after all compared policies pass undisturbed screening; (2) generate the post-disturbance state once using a policy-free 0.4-s end-effector controller program; (3) restore that same physical release for all policies; and (4) compare recovery by six signed task-frame directions × three magnitudes (16, 32, 48 mm), retaining both condition-wise differences and an aggregate score with declared weights.

A condition is a direction–magnitude pair. Its recovery profile is the probability of reaching the next task milestone within 6 s; the policy-pair contrast is the difference between their recovery probabilities at the same release. The aggregate score is an explicitly weighted summary of these conditional probabilities, not a universal robustness rank. Each policy is freshly queried with its prior action queue cleared, and retains its own native execution cadence.

At one book-to-caddy state, SmolVLA and GR00T have nearly identical original aggregate scores (0.761 vs. 0.750), but 20 new rollouts per condition show 20/20 vs. 0/20 successes after one 48-mm release and 8/20 vs. 20/20 after another. This is why the conditional contrast matters, even when a scalar score does not separate the policies.

The old symbols u and h referred to a separate, one-control 5-mm end-effector probe and a subsequent four-control observation window. That ancillary local-response analysis and its propositions have been removed from the revised main argument. The paper now defines the retained state, condition, release, recovery profile, policy contrast, and aggregate score before presenting Results.

We have clarified the method and notation in §3.1–3.2 (pp. 3–5; Figure 2, p. 4), the motivating example in Figure 1 (p. 2), and the prior-work context in §2 (p. 3). Further details are in Appendices A.1–A.3 (pp. 14–22) and B.1 (pp. 22–23).

[W2 / W4 / W6] Start-State Coverage, Screening Bias, and Policy Variability. Overall, since all the experiments are in simulation.

Why cannot we do more rollouts from a larger pool of start state distributions even for subtasks.

Couldn’t some of the policies be biased towards these fixed states?

It would be interesting to compare how this approach compares against running a lot more rollouts from intermediate states without this structured perturbation.

Specifically, I think it’s quite well known that VLAs have high variance especially some of the academic ones.

We extended the evaluation beyond the original 20 clean-screened states with 62 distinct new demonstration starts from 33 existing LIBERO tasks (sets A and B). These are new states, not 33 new tasks; sets A and B share 29 task identities but no saved states. We evaluate a fourth released policy, π0.5, in separate matched comparisons. Across all sets, at least one policy is evaluated on 39 of the 40 LIBERO tasks, but there is no four-policy global ranking because the four policies do not share a common state set.

Table R1. Large recovery gaps across matched evaluation sets (6-s milestone).

EvaluationMatched pairStates with a success-rate gap ≥0.6Rollout-noise expectationStatistical result
Original panel (7 tasks)SmolVLA–GR00T18/206.7Bonferroni-adjusted p = 1.02×10⁻⁹; task-level p = 0.003
New starts A (30 states)OpenVLA-OFT–GR00T21/3015.5p = 0.002
New starts A (29 jointly eligible states)π0.5–GR00T15/297.0Holm p = 6.0×10⁻⁴
New starts B (32 other states)π0.5–GR00T17/327.4p = 2.2×10⁻⁶

These comparisons are tested separately, not combined into a single prevalence or policy ranking. The original any-policy-pair 18/20 count is not significant (null expectation 17.4, p = 0.428); the supported original claim is specifically for SmolVLA–GR00T.

We also performed 40 new undisturbed rollouts per stochastic policy at each of the 20 original states. Of 1,600 rollouts, 1,595 reached both endpoints. On the 18 states where both policies passed 40/40, 16/18 retained an original disturbed gap ≥0.6 (null 6.4, Bonferroni p = 4.5×10⁻⁸). This is a retrospective stricter-screen check, not a new disturbed replicate. A conservative test allowing uncertain underlying clean-success rates does not reject a clean-failure explanation (Berger–Boos p = 0.12); our comparisons remain conditional on screening.

To separate real structure from high rollout-to-rollout variance, we also re-tested 28 previously selected conditions from identical releases using 20 wholly new rollouts per stochastic policy: 28/28 reproduced a gap above 0.5 at 6 s (largest Holm-adjusted p = 0.016; a paired sensitivity check retained 27/28). A separate retrospective prediction study on 79 states from 31 tasks found Brier loss 0.0235 for the recovery profile vs. 0.1094 for a scalar score, using different test rollouts from the same releases. These are distinct forms of evidence: the 28-condition study confirms selected gaps, whereas the prediction study tests information retained at already measured states.

We did not run an unrestricted random-intermediate-state baseline. Indeed, 5 of 33 candidate states in the physical-control study showed a substantial policy gap without any disturbance, illustrating why a jointly clean screen is important for comparing recovery rather than baseline task competence.

We report the evaluation sets in §4 / Table 1 (p. 5), the replication and re-screening results in §5.1 / Figure 3 (pp. 6–7), and the prediction study in §5.3 / Table 2 (pp. 8–9). Supporting analyses are in §3.1 (pp. 3–4) and Appendices C.1–C.4 (pp. 23–29), E.2 (pp. 33–34), and F.3 (pp. 39–40).

[W3 / W5 / W7] Task-Frame Directions, Action Noise, and Joint-Space Alternatives. The local perturbations definitely are helpful but does a structured perturbation (e.g. having a vector in the gravity plane) really matter that much.

Another, baseline that we often use is is to inject arbitrary noise in the actions (further away from goal) and decay that noise.

How does that compare?

But while this paper focuses on an important problem, I think the lack of comparison on why their perturbation approach is better is a bit unclear to me.

I guess another problem would be perturbations in joint space vs end-effector space.

The paper only focuses on end-effector perturbations, but such perturbations are clearly biased since some joints may not change at all.

Hence, joint space perturbations can be more information and also more challenging to do.

We compared task-frame directions with random directions and evaluated a same-duration controller takeover control and alternative perturbation programs. The results of these distinct comparisons are summarized below.

Table R2. Controlled comparisons with alternative disturbance programs.

ComparisonWhat we testedMain outcome
Task-frame vs. random directionsSix task-frame and six random 48-mm directions at 49 new states from 23 tasks, with 20 new confirmation rollouts per selected conditionBoth expose reproducible gaps (mean new gap 0.69 vs. 0.59); task-averaged task-frame minus random gap 0.077 [−0.029, 0.189], p = 0.199. Neither superiority nor equivalence is established.
Translation vs. same-duration controller takeoverA 48-mm program versus a 0.4-s zero-command program on new states from 25 tasks; 20 rollouts per policy and programSmolVLA: −0.34 [−0.53, −0.16], Holm p = 0.003; GR00T: −0.11 [−0.24, 0.01], p = 0.13 (not significant). The zero-command controller still moves the hand (median 15 mm), so this is a program-level contrast, not an isolated force or displacement effect.
Decaying action-noise proxyPhysical-comparability pilot against 16-mm end-effector releasesOnly 6/17 candidate states met the comparability criterion; the proxy could not attain 48 mm (median achieved offset at most 22 mm). No policy-outcome action-noise comparison was run.
Joint-position vs. end-effector programsAt 14 states from 14 tasks with approximately matched 16-mm hand offsets, each program’s release was shared across policiesAt the primary 6-s endpoint, 1,120/1,120 disturbed rollouts succeeded, leaving no policy gap under either program (a ceiling). The study establishes neither superiority nor equivalence. The programs do not have identical full physical states.

The purpose of the task frame is standardization, reproducibility, and interpretability: each direction has a consistent task-relative meaning and each policy is tested from the same release. We do not claim that task-frame perturbations are inherently harder or more informative than random directions, decaying noise, or joint-space programs.

We have added these control and alternative-perturbation results to §5.4 (pp. 9–10; Figure 5, p. 9) and Appendices F.1 (pp. 37–38), F.3 (pp. 39–40), and F.4 / Table F.2 (pp. 40–41). The random-direction comparison is summarized on p. 10.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised manuscript and Supplement

We sincerely thank the reviewers for their constructive feedback. We have uploaded a revised manuscript and supplementary materials.

  • The most substantial changes address presentation and clarity. In response to the reviewers' feedback, we reorganized the manuscript, redesigned the figures, clarified the methodology and experimental settings, and aligned the results with their supporting appendices. We also conducted targeted additional experiments and analyses to address questions concerning clean screening, independent confirmation, evaluation breadth, predictive informativeness, and physical controls. The revised manuscript explicitly documents important corrections to the original statistical interpretation.

  • The core research question, evaluation protocol, and main findings remain unchanged. The revision retains our original matched-release evaluation, recovery maps, aggregate score, and primary LIBERO experiments. We now refer to the method as State-Matched Recovery Profiling (SMaRP), with the recovery profile corresponding to the original disturbance-tolerance map.

  • We particularly encourage reviewers to watch the three short supplementary videos, which illustrate the method and findings through actual LIBERO simulation replays:

  1. Video 1 — Map Construction: How recovery maps are constructed.
  2. Video 2 — Large Policy Differences: How two policies exhibit opposite recovery outcomes under different conditions.
  3. Video 3 — Six Directions Radial Map: How recovery varies across six disturbance directions.

We provide detailed, question-by-question responses with precise manuscript references under each review. We hope the revised presentation and additional validation make the contribution and supporting evidence substantially clearer.

Area_Chair_jLYZOpenReview ↗

Reminder: Please Follow Up on the Author Rebuttal

Dear Reviewers,

As we are approaching the end of the discussion period, I would like to remind you to review the authors' rebuttal and assess whether your concerns have been adequately addressed.

If you have any remaining concerns, please raise them as soon as possible to allow the authors sufficient time to respond. If your concerns have been resolved, please leave a brief acknowledgment to let the authors know that you have read and considered their responses. Thank you for your time and efforts in the review process.

Best regards, Area Chair

We have made a few refinements to our earlier responses and reorganized them into separate, issue-focused comments to make them easier to follow. Each comment now directly addresses the relevant reviewer concerns, presents the supporting experiments and key quantitative results, and points to the corresponding sections of the revised manuscript.

We would greatly appreciate it if the reviewers could revisit our updated responses and revised manuscript before the discussion period closes.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.