- Rating
- 3
- Confidence
- 4(+2)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates why VLA acceleration methods can appear both faster and more successful results in simulation benchmarks.
It identifies the causes by analyzing the relationships among multiple evaluation factors.
The contribution lies in proposing a diagnostic workflow and a set of benchmark repairs.
Critical strengths
The problem is important and valuable, and has puzzled many researchers. Although some researchers attribute these results to the benchmarks being too simple, the underlying causes may be more complex.
The experiment was based on seven benchmarks, making it relatively comprehensive and broad coverage from this perspective
The conclusions are clear and well presented. Direct comparisons before and after the fixes demonstrate how evaluation conditions can change the rankings of the same methods as evaluation conditions vary.
Critical weaknesses
The number of models used is somewhat limited; π0.5 is the primary one. In fact, there are many other common models, such as FastWAM;
Despite the coverage of seven benchmarks, in fact, the total number of tasks covered in the experiments is relatively limited.
Questions for authors
Please address the weaknesses above, or explain more about it
6 comments
Thank you to the authors for providing additional results across models and benchmarks. I find this work novel and valuable; the specific examples clearly demonstrate how evaluation settings can influence method comparisons. Further discussion and clearer organization would help readers understand the research findings and reuse the diagnostic methods.
- Training–Test Distribution Shift
When physical parameters such as object mass change while the base model remains unchanged, changes in performance and rankings may reflect both the limitations of the original evaluation setup and the sensitivity of different inference methods to training–test distribution shifts. How do the authors distinguish between these two explanations? The Place Fan experiment, which uses base models fine-tuned on examples collected under modified settings, provides helpful evidence. Please discuss this point in light of these results, as it may affect the interpretation and scope of the core conclusion that acceleration gains may stem from benchmark flaws.
- Criteria for Selecting Revised Evaluation Settings
For clear implementation errors, such as reading the wrong variables, the basis for correction is relatively straightforward. However, for settings that allow for significant flexibility, such as object quality and success thresholds, how can we determine which revised values are more appropriate as evaluation criteria? When multiple settings are all reasonable, how should we interpret the paper’s conclusions? Although this is an open-ended question, clarifying the criteria for selection and their implications helps to clarify the paper’s main contributions.
- Connecting modification types to conclusions
Building on the existing taxonomy, we recommend making a clearer distinction between modifications that successfully correct identified errors, those that alter the environment or the distribution of instructions, and those that introduce additional evaluation criteria such as smoothness. Explaining which conclusions each category of results supports will help readers understand how these modifications improve existing benchmarks and how they affect method comparisons.
- Systematic diagnosis and transferability
The Organize Table case in RoboDojo demonstrates that even relatively new benchmarks may suffer from inconsistencies between instructions and the initial scenario. Could the authors further integrate the existing classifications, the workflow described in Section 4, and the recommendations in Appendix D to explain at which stages of benchmark construction these issues arise and how they can be systematically identified and verified? Discussing how this workflow can be applied to the evaluation of strategies beyond acceleration methods would help clarify its value as a transferable diagnostic method and make it easier to understand and reuse.
Response to Reviewer (Part 1/2):
We thank the reviewer for recognizing the novelty and practical value of our work and for the specific suggestions on interpreting and reusing the findings. We clarify the interpretation, rationale, and reuse of our findings below.
Q1. Training–test distribution shifts and limitations of the evaluation settings
We agree that changing physical parameters with a fixed model may affect performance through both limitations of the original settings and differences in methods' sensitivity to training–test distribution shifts.
We collected new demonstrations and fine-tuned the model under the revised Place Fan settings; see Appendix A for the training and comparison protocol. Results associated with §7.3.2 are shown below (success rate, %):
| Condition | Baseline | DP-Cache-Slow | DP-Cache-Fast | ProbeFlow |
|---|---|---|---|---|
| A: Original benchmark | 40 | 51 | 61 | 59 |
| B: Revised object masses | 38 | 38 | 33 | 34 |
| C: Revised object masses, with fine-tuning under these settings | 53 | 49 | 45 | 42 |
A vs. B shows that, with a fixed checkpoint, the three accelerated methods' advantages over baseline disappear or become deficits after revising the physical settings.
B vs. C further shows that fine-tuning on data matching the test distribution improves all four methods, indicating that the original training–test distribution shift affected model performance.
A vs. C shows that the original gains from training-free acceleration are not restored after fine-tuning, indicating that limitations of the original settings affect assessments of where acceleration methods are suitable.
Physical realism and training adaptation are not mutually exclusive considerations: the former concerns whether parameters match real objects, while the latter affects performance under those dynamics. We will clarify both in the revision.
Q2. Rationale for revised evaluation settings
Masses follow real-world references. Fig. 4(c) shows RoboTwin's hammer at 1 g versus a 600 g real-world reference, and LIBERO's plate at 11.52 g versus 150 g. We target large discrepancies to improve simulation's relevance to real operations: visual realism alone does not ensure realistic dynamics. Where multiple values suit the target scenario, we recommend environment randomization within physically grounded ranges and comparisons on matched task instances, assessing adaptation to physical variation and reducing dependence on a single setting.
Success criteria follow semantic consistency. Appendix B and Appendix D, Advice #1, guide checks of agreement among instructions, demonstrations, and checkers on target objects, required actions, and completion conditions:
-
Place Shoe: Remove the unrequested toe-orientation constraint; retain the instructed mat-placement check.
-
Place Bread Skillet: Align the specified arm with demonstrations and add the corresponding arm-use check.
-
Select Mahjong: Retain the instructed pickup requirement and remove the unrequested placement condition.
Thresholds follow task requirements and scene measurements; sensitivity analysis addresses reasonable alternatives.
For Put Bottles Dustbin, final bottle positions alone do not verify the instructed pickup-and-delivery process. Appendix B reports measured fall times into the bin from positions near the table edge, at heights from tabletop level to 30 cm above it. These measurements informed a 250 ms lookback window: within this window before each bottle enters the checker region, the gripper holding that bottle must have appeared directly above the region. Thus, the added process criterion has a task-specific measurement basis.
For semantic-consistency repairs, reported success rates retain the original thresholds for remaining spatial conditions. Where instructions such as “center” or “near” leave exact tolerances unspecified, multiple values may be reasonable. We separately examine threshold sensitivity using Scan Object (§6.1(2); Fig. 4(b)): fix the axial-distance threshold, vary the transverse-alignment tolerance, and measure each method's success rate as the acceptance region changes. This reveals dependence on threshold choices rather than prescribing a universally correct value; applicability should be interpreted against the target application's precision requirements. We will clarify these threshold-selection rationales and their implications in the revision.
We thank the reviewer for recognizing the importance of the problem and the value of our cross-benchmark analysis. To address model and task coverage, we added experiments across models and acceleration methods using OpenVLA and FastWAM, and extended our audit to Meta-World. We respond below.
W1 & Q1. The number of models is limited; π0.5 is the primary one, while other common models include FastWAM.
We added two model families: autoregressive OpenVLA with training-free VLA-Cache, and FastWAM with DP-Cache-Fast, DP-Cache-Slow, and ProbeFlow. We evaluated them before and after correcting issues.
OpenVLA. The first two tasks fix BDDL instruction delivery in LIBERO-PRO; Two-Mug Placement revises object mass in LIBERO; Place Shoe aligns RoboTwin's checker with the instruction. Entries show success rates (%, before→after).
FastWAM. We added comparisons on RoboTwin tasks involving process checks, instruction consistency, and physical settings.
The effects extend beyond π0.5 to autoregressive and WAM models with different training-free acceleration methods. Their magnitude and direction vary: on Two-Mug Placement, VLA-Cache changes from 8 percentage points above baseline to 14 below; on Put in Cabinet, some FastWAM variants narrow their gap to baseline. Our goal is to identify evaluation issues that affect method comparisons, enabling more accurate judgments of acceleration gains and suitable application scenarios.
W2 & Q1. Despite covering seven benchmarks, the total number of tasks in the experiments is relatively limited.
Original coverage and audit expansion. Appendix B documents 22 bugs, their affected settings, and repairs; Appendix C provides additional experiments beyond the main text. The main and appendix tables report 32 tasks/subsets, including 10 obtained by extending the audit based on identified failure mechanisms.
We classify issues into task consistency, initialization, reproducibility, and design limitations, using trajectory and checker acceptance-region plots to aid diagnosis. As reviewer Q9eB recognized, this is a "cheap, reusable audit practice." Following §4, we inspect tasks sharing relevant components, extend the audit to "tasks without anomalous gains," and verify each case. This classification and diagnostic workflow can be applied to other benchmarks to identify similar issues and assess suitable application scenarios for acceleration methods more objectively and reliably. This workflow supports more reliable assessment of where acceleration methods are suitable; we further apply it to Meta-World below.
Meta-World audit. Applying this workflow to Meta-World, a benchmark outside our original evaluation, identified seven issues: five task-consistency issues, one initialization issue, and one design limitation. These involve success checks inconsistent with current states, target objects, or instructions; incorrect evaluation-quota accounting; and acceptance of incomplete operations.
Meta-World provides two reward-function implementations, V1 and V2, whose intermediate quantities are also used by the success checker.
We added before/after evaluations for two tasks:
Hand Insert: The V1 branch checker returns the object's initial distance to the target, rather than its current distance. Instances initially beyond the threshold therefore cannot pass this condition during execution. We correct this distance calculation.
Peg Insert Side: V1 checker uses the grasp reference point's distance to the target, whereas V2 uses a weighted peg-head distance, potentially assigning different labels to the same pose. We align the former with V2's peg-head criterion.
Below are π0.5 success rates (%) from scoring the same executed trajectories before and after correction.
The original checkers assign all methods 0% and 14%, respectively; corrected rates are 74–80% and 88–96%. The corrections change measured success and reveal method differences previously hidden by identical scores. These experiments expand task and benchmark coverage and demonstrate that the workflow identifies similar issues in new settings. We will include the cross-model and cross-benchmark findings, correction rationales, and results in the revision.