- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates how VLA policies recover from controlled disturbances during manipulation. It proposes disturbance-tolerance maps to reveal behavioral differences that ordinary task-success rates can hide.
Critical strengths
-
Evaluating recovery and robustness beyond nominal task success is an essential component on understanding robot policy performances.
-
The experiments compare three policies under matched disturbances and include repeated trials and controls for execution cadence.
Critical weaknesses
- The writing and organization make the experimental setup difficult to follow. The relationships between local-response probes, tolerance maps, and aggregate scores are not explained clearly enough.
- The manuscript introduces several experimental panels without a concise overview of their purposes and connections. Essential protocol details are scattered across the appendices.
- The theoretical propositions establish general possibilities, but their practical contribution to the proposed evaluation is insufficiently demonstrated.
- The central evidence is limited to three policies in LIBERO simulation. The headline result of differences at 18/20 anchors is conditional on a small subset where all policies pass the clean-success screen, which limits its generality.
- The introduction contains no citations to support its motivation or position the contribution relative to prior work on robustness and recovery evaluation. This makes the research gap and novelty difficult to assess.
- How can a ICLR submission cite other ICLR submissions? Also, both citations seem not to be relevant with the theme of the paper.
Questions for authors
The paper needs substantial reorganization and clearer writing. Present the complete evaluation protocol first, explain each experimental panel in one place, and distinguish the main tolerance-map contribution from the supporting local-response analysis. State the conclusions more narrowly around the tested policies and disturbances.
6 comments
[W3] The Role and Practical Meaning of the Remaining Lemma. The theoretical propositions establish general possibilities, but their practical contribution to the proposed evaluation is insufficiently demonstrated.
We removed the local-response propositions, which were not necessary to the central measurement result. The revised paper retains one simple dominance lemma directly linked to the score–profile relationship. Specifically, if policy A recovers at least as often as policy B in every shared condition, A's aggregate score is at least as high under any weighting over those conditions. Conversely, if A leads in some conditions and B leads in others, a ranking by one weighted average is not universal.
This is the reason we report the complete recovery profile and pairwise contrasts alongside the aggregate: they show which release conditions drive a score and whether the ordering depends on the evaluator's weighting. The lemma is not presented as a novel general theory.
At the book-to-caddy state, the original five-rollout maps yield aggregate scores 0.761 (SmolVLA) and 0.750 (GR00T), but new rollouts from two separate 48-mm releases give 20/20 vs. 0/20 and 8/20 vs. 20/20. Both policies therefore lead in a measured condition. The empirical reversal makes clear why the aggregate ranking is not invariant to the conditions emphasized.
We present the score–profile lemma in §3.2 (pp. 4–5), its concrete interpretation in §5.2 (pp. 7–8; Figure 1, p. 2), and its proof and fresh-rollout illustration in Appendices A.3 (p. 22) and D.1 (pp. 30–31).
[W4 / Q] Screened Subset, Policy Breadth, and Correct Inference Scope. The central evidence is limited to three policies in LIBERO simulation. The headline result of differences at 18/20 anchors is conditional on a small subset where all policies pass the clean-success screen, which limits its generality.
State the conclusions more narrowly around the tested policies and disturbances.
We have clarified that the headline 18/20 result is conditional on a joint undisturbed screen. It concerns 20 admitted states from 7 tasks, not 20 independent tasks, and tests the SmolVLA–GR00T pair specifically. Its pair-specific rollout-noise expectation is 6.7 (Bonferroni p = 1.02×10⁻⁹), with all 7/7 task roots above their null (p = 0.003). The earlier any-pair 18/20 interpretation was corrected: its null is 17.4 and p = 0.428, so it is not evidential.
To assess stability, we re-screened the 20 original states using 40 new undisturbed rollouts per stochastic policy (1,595/1,600 successes at both endpoints). Among 18 states passing 40/40 for both policies, 16 retain an original gap ≥0.6 (null 6.4, p = 4.5×10⁻⁸). The conservative clean-failure test remains non-rejecting (Berger–Boos p = 0.12), so we do not infer identical underlying clean-success probabilities.
The breadth of evidence is now greater, but comparisons retain their design boundaries:
Table R7. Additional evidence and the limits of the tested comparisons.
| Additional evidence | Result |
|---|---|
| 62 nonoverlapping new demonstration states from 33 existing tasks | OFT–GR00T 21/30; π0.5–GR00T 15/29 in set A and 17/32 in set B (separate comparisons) |
| Independent confirmation at selected releases | 28/28 6-s gaps reproduced on 20 new rollouts per stochastic policy; 21/28 at 11 s |
| Model coverage | Four released policies in total, but no state shared by all four and no direct SmolVLA–π0.5 pair |
| Scope | LIBERO simulation, saved pick and transport states; not a real-robot or dexterous-manipulation result |
Thus, the supported conclusion is that large condition-wise recovery differences are measurable and reproducible within these tested, jointly screened comparisons. We do not claim a four-policy ranking, gap prevalence across arbitrary new tasks, or direct deployment benefit.
We have clarified the scope and inference units in §4 / Table 1 (p. 5), §5.1 / Figure 3 (pp. 6–7), Discussion §6 (p. 10), and Appendices C.1–C.5 (pp. 23–30).
We sincerely thank the reviewer for the thoughtful and constructive feedback. We have clarified the evaluation protocol, its theoretical motivation, experimental scope, and relationship to prior work, as detailed below.
We reorganized Method §3 to explain the complete measurement in one sequence, illustrated by Figure 2: (1) choose a saved pick/transport state jointly passed in undisturbed screening; (2) create a policy-free, 0.4-s end-effector translation along six task-frame directions and magnitudes 16/32/48 mm; (3) save the resulting release and restore it identically for each policy; (4) resume each policy with an empty action queue and a fresh query; (5) report the condition-wise recovery profile, the policy-pair difference, and an aggregate score under declared evaluation weights.
A recovery profile maps each physical condition to the probability of reaching the next milestone within 6 s; a contrast subtracts the profiles of two policies starting from the same release; an aggregate score averages one profile under an explicitly chosen condition distribution. The separate local-response probe and its propositions have been removed from the main argument.
Table R6. Research questions, main findings, and corresponding results.
All evaluation sets and rollout counts are described together in §4 / Table 1 (p. 5) rather than dispersed across result sections. Appendices A and C give reproducible protocol and study details; Appendices D–F follow the main-results order. This layout separates primary evidence, selected-condition confirmation, retrospective prediction, and exploratory controls. Three short supplementary replay videos illustrate map construction, policy differences and the six-direction map; they are visualizations of logged simulation actions, not new experimental samples.
We have reorganized the protocol in §3 / Figure 2 (pp. 3–5), consolidated the experiments in §4 / Table 1 (p. 5), and organized the results around Figures 1 and 3–5 (pp. 2, 6–9) and Table 2 (p. 8). Technical and experimental details are in Appendices A (pp. 14–22), C (pp. 23–30), and D–F (pp. 30–41).