acceptodds
Under review as a conference paper at ICLR 2027

Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration

Abstract

Simulated manipulation benchmarks are the standard tool for evaluating vision-language-action (VLA) policies and the acceleration methods that reduce their inference latency for on-robot deployment. On these benchmarks, we observe that some training-free acceleration methods, which approximate the baseline policy's computation, achieve higher measured success rates than the baseline itself. Success rates alone cannot establish whether such gains come from better task execution or from evaluation flaws. We therefore investigate two kinds of benchmark flaws behind these gains: bugs, where the implementation does not match the intended task or evaluation protocol, and design limitations, where success criteria and simulation settings do not fully capture how acceleration affects task execution. Starting from tasks with anomalous gains, we localize root causes by plotting object trajectories against checker acceptance regions, and classify the resulting bugs into task consistency, initialization, and reproducibility. Extending this audit to seven benchmarks, including RoboTwin, LIBERO-Plus, and VLABench, we identify 22 bugs of these types and 4 design limitations. For the latter, we revise permissive success checkers, correct unrealistic object masses, and add a motion-aware score that favors smoother actions. Experiments show that bug fixes can reverse method rankings, moving the baseline from last to first on one task. Addressing design limitations can likewise remove anomalous gains: on another task, the baseline moves from 21 percentage points behind an accelerated method to 5 points ahead. Gains attributed to acceleration can therefore be artifacts of the benchmark rather than better task execution. We release our bug fixes and revised benchmark settings to support trustworthy evaluation of VLA acceleration.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, Inference acceleration, Benchmark and Evaluation reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cPQROpenReview ↗
Rating
3
Confidence
4(+2)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates why VLA acceleration methods can appear both faster and more successful results in simulation benchmarks.

It identifies the causes by analyzing the relationships among multiple evaluation factors.

The contribution lies in proposing a diagnostic workflow and a set of benchmark repairs.

Critical strengths

The problem is important and valuable, and has puzzled many researchers. Although some researchers attribute these results to the benchmarks being too simple, the underlying causes may be more complex.

The experiment was based on seven benchmarks, making it relatively comprehensive and broad coverage from this perspective

The conclusions are clear and well presented. Direct comparisons before and after the fixes demonstrate how evaluation conditions can change the rankings of the same methods as evaluation conditions vary.

Critical weaknesses

The number of models used is somewhat limited; π0.5 is the primary one. In fact, there are many other common models, such as FastWAM;

Despite the coverage of seven benchmarks, in fact, the total number of tasks covered in the experiments is relatively limited.

Questions for authors

Please address the weaknesses above, or explain more about it

6 comments

We thank the reviewer for recognizing the importance of the problem and the value of our cross-benchmark analysis. To address model and task coverage, we added experiments across models and acceleration methods using OpenVLA and FastWAM, and extended our audit to Meta-World. We respond below.

W1 & Q1. The number of models is limited; π0.5 is the primary one, while other common models include FastWAM.

We added two model families: autoregressive OpenVLA with training-free VLA-Cache, and FastWAM with DP-Cache-Fast, DP-Cache-Slow, and ProbeFlow. We evaluated them before and after correcting issues.

OpenVLA. The first two tasks fix BDDL instruction delivery in LIBERO-PRO; Two-Mug Placement revises object mass in LIBERO; Place Shoe aligns RoboTwin's checker with the instruction. Entries show success rates (%, before→after).

TaskBaselineVLA-Cache
Bowl Placement On Stove0→860→78
Bowl Placement Next to Plate0→400→32
Two-Mug Placement48→3656→22
Place Shoe0→44→4

FastWAM. We added comparisons on RoboTwin tasks involving process checks, instruction consistency, and physical settings.

TaskBaselineDP-Cache-FastDP-Cache-SlowProbeFlow
Put in Dustbin90→9089→8791→8889→87
Place Shoe93→9793→9492→9293→94
Put in Cabinet92→9288→9287→8784→86
Place Fan94→8384→7294→8093→80

The effects extend beyond π0.5 to autoregressive and WAM models with different training-free acceleration methods. Their magnitude and direction vary: on Two-Mug Placement, VLA-Cache changes from 8 percentage points above baseline to 14 below; on Put in Cabinet, some FastWAM variants narrow their gap to baseline. Our goal is to identify evaluation issues that affect method comparisons, enabling more accurate judgments of acceleration gains and suitable application scenarios.

W2 & Q1. Despite covering seven benchmarks, the total number of tasks in the experiments is relatively limited.

Original coverage and audit expansion. Appendix B documents 22 bugs, their affected settings, and repairs; Appendix C provides additional experiments beyond the main text. The main and appendix tables report 32 tasks/subsets, including 10 obtained by extending the audit based on identified failure mechanisms.

We classify issues into task consistency, initialization, reproducibility, and design limitations, using trajectory and checker acceptance-region plots to aid diagnosis. As reviewer Q9eB recognized, this is a "cheap, reusable audit practice." Following §4, we inspect tasks sharing relevant components, extend the audit to "tasks without anomalous gains," and verify each case. This classification and diagnostic workflow can be applied to other benchmarks to identify similar issues and assess suitable application scenarios for acceleration methods more objectively and reliably. This workflow supports more reliable assessment of where acceleration methods are suitable; we further apply it to Meta-World below.

Meta-World audit. Applying this workflow to Meta-World, a benchmark outside our original evaluation, identified seven issues: five task-consistency issues, one initialization issue, and one design limitation. These involve success checks inconsistent with current states, target objects, or instructions; incorrect evaluation-quota accounting; and acceptance of incomplete operations.

Meta-World provides two reward-function implementations, V1 and V2, whose intermediate quantities are also used by the success checker.

We added before/after evaluations for two tasks:

  1. Hand Insert: The V1 branch checker returns the object's initial distance to the target, rather than its current distance. Instances initially beyond the threshold therefore cannot pass this condition during execution. We correct this distance calculation.

  2. Peg Insert Side: V1 checker uses the grasp reference point's distance to the target, whereas V2 uses a weighted peg-head distance, potentially assigning different labels to the same pose. We align the former with V2's peg-head criterion.

Below are π0.5 success rates (%) from scoring the same executed trajectories before and after correction.

TaskBaseline (NFE10)DP-Cache-SlowDP-Cache-FastProbeFlow
Hand Insert0→740→780→800→80
Peg Insert Side14→9614→9414→9214→88

The original checkers assign all methods 0% and 14%, respectively; corrected rates are 74–80% and 88–96%. The corrections change measured success and reveal method differences previously hidden by identical scores. These experiments expand task and benchmark coverage and demonstrate that the workflow identifies similar issues in new settings. We will include the cross-model and cross-benchmark findings, correction rationales, and results in the revision.

Thank you to the authors for providing additional results across models and benchmarks. I find this work novel and valuable; the specific examples clearly demonstrate how evaluation settings can influence method comparisons. Further discussion and clearer organization would help readers understand the research findings and reuse the diagnostic methods.

  1. Training–Test Distribution Shift

When physical parameters such as object mass change while the base model remains unchanged, changes in performance and rankings may reflect both the limitations of the original evaluation setup and the sensitivity of different inference methods to training–test distribution shifts. How do the authors distinguish between these two explanations? The Place Fan experiment, which uses base models fine-tuned on examples collected under modified settings, provides helpful evidence. Please discuss this point in light of these results, as it may affect the interpretation and scope of the core conclusion that acceleration gains may stem from benchmark flaws.

  1. Criteria for Selecting Revised Evaluation Settings

For clear implementation errors, such as reading the wrong variables, the basis for correction is relatively straightforward. However, for settings that allow for significant flexibility, such as object quality and success thresholds, how can we determine which revised values are more appropriate as evaluation criteria? When multiple settings are all reasonable, how should we interpret the paper’s conclusions? Although this is an open-ended question, clarifying the criteria for selection and their implications helps to clarify the paper’s main contributions.

  1. Connecting modification types to conclusions

Building on the existing taxonomy, we recommend making a clearer distinction between modifications that successfully correct identified errors, those that alter the environment or the distribution of instructions, and those that introduce additional evaluation criteria such as smoothness. Explaining which conclusions each category of results supports will help readers understand how these modifications improve existing benchmarks and how they affect method comparisons.

  1. Systematic diagnosis and transferability

The Organize Table case in RoboDojo demonstrates that even relatively new benchmarks may suffer from inconsistencies between instructions and the initial scenario. Could the authors further integrate the existing classifications, the workflow described in Section 4, and the recommendations in Appendix D to explain at which stages of benchmark construction these issues arise and how they can be systematically identified and verified? Discussing how this workflow can be applied to the evaluation of strategies beyond acceleration methods would help clarify its value as a transferable diagnostic method and make it easier to understand and reuse.

AuthorsOpenReview ↗

Response to Reviewer (Part 1/2):

We thank the reviewer for recognizing the novelty and practical value of our work and for the specific suggestions on interpreting and reusing the findings. We clarify the interpretation, rationale, and reuse of our findings below.

Q1. Training–test distribution shifts and limitations of the evaluation settings

We agree that changing physical parameters with a fixed model may affect performance through both limitations of the original settings and differences in methods' sensitivity to training–test distribution shifts.

We collected new demonstrations and fine-tuned the model under the revised Place Fan settings; see Appendix A for the training and comparison protocol. Results associated with §7.3.2 are shown below (success rate, %):

ConditionBaselineDP-Cache-SlowDP-Cache-FastProbeFlow
A: Original benchmark40516159
B: Revised object masses38383334
C: Revised object masses, with fine-tuning under these settings53494542

A vs. B shows that, with a fixed checkpoint, the three accelerated methods' advantages over baseline disappear or become deficits after revising the physical settings.

B vs. C further shows that fine-tuning on data matching the test distribution improves all four methods, indicating that the original training–test distribution shift affected model performance.

A vs. C shows that the original gains from training-free acceleration are not restored after fine-tuning, indicating that limitations of the original settings affect assessments of where acceleration methods are suitable.

Physical realism and training adaptation are not mutually exclusive considerations: the former concerns whether parameters match real objects, while the latter affects performance under those dynamics. We will clarify both in the revision.

Q2. Rationale for revised evaluation settings

Masses follow real-world references. Fig. 4(c) shows RoboTwin's hammer at 1 g versus a 600 g real-world reference, and LIBERO's plate at 11.52 g versus 150 g. We target large discrepancies to improve simulation's relevance to real operations: visual realism alone does not ensure realistic dynamics. Where multiple values suit the target scenario, we recommend environment randomization within physically grounded ranges and comparisons on matched task instances, assessing adaptation to physical variation and reducing dependence on a single setting.

Success criteria follow semantic consistency. Appendix B and Appendix D, Advice #1, guide checks of agreement among instructions, demonstrations, and checkers on target objects, required actions, and completion conditions:

  • Place Shoe: Remove the unrequested toe-orientation constraint; retain the instructed mat-placement check.

  • Place Bread Skillet: Align the specified arm with demonstrations and add the corresponding arm-use check.

  • Select Mahjong: Retain the instructed pickup requirement and remove the unrequested placement condition.

Thresholds follow task requirements and scene measurements; sensitivity analysis addresses reasonable alternatives.

For Put Bottles Dustbin, final bottle positions alone do not verify the instructed pickup-and-delivery process. Appendix B reports measured fall times into the bin from positions near the table edge, at heights from tabletop level to 30 cm above it. These measurements informed a 250 ms lookback window: within this window before each bottle enters the checker region, the gripper holding that bottle must have appeared directly above the region. Thus, the added process criterion has a task-specific measurement basis.

For semantic-consistency repairs, reported success rates retain the original thresholds for remaining spatial conditions. Where instructions such as “center” or “near” leave exact tolerances unspecified, multiple values may be reasonable. We separately examine threshold sensitivity using Scan Object (§6.1(2); Fig. 4(b)): fix the axial-distance threshold, vary the transverse-alignment tolerance, and measure each method's success rate as the acceptance region changes. This reveals dependence on threshold choices rather than prescribing a universally correct value; applicability should be interpreted against the target application's precision requirements. We will clarify these threshold-selection rationales and their implications in the revision.

Reviewer_2vuKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This article empirically examines current benchmarks for VLA model acceleration algorithms. The authors found that some training-free acceleration methods, while seemingly achieving unusually high success rates, actually exploit vulnerabilities in the testing environment. Through in-depth audits of seven mainstream benchmarks, the study uncovered 22 code vulnerabilities and four design limitations. After fixing these benchmark flaws and introducing more rigorous evaluation, the model rankings dramatically reversed: the baseline model's real-world performance rebounded significantly, even surpassing the accelerated algorithms.

Critical strengths
  1. This systematic and exhaustive empirical audit goes beyond a single platform. It conducts a meticulous code-level review of seven mainstream VLA operating benchmarks (RoboTwin, LIBERO, VLABench, etc), identifying 22 code bugs and 4 design flaws. This cross-framework summary (Task consistency, Initialization, Reproducibility) provides the community with a valuable guide to avoiding common pitfalls.

  2. This paper does not stop at the level of "criticism" but proposes new and practical metrics to address existing design flaws: for example, introducing motion-aware scoring and correcting absurd object masses in simulation environments.

  3. The authors propose a workflow that can be used for future benchmarks examination.

Critical weaknesses
  1. The Paper lacks rigorous theoretical boundaries, failing to mathematically define when the algorithm's approximation error constitutes beneficial implicit regularization and when it degenerates into high-frequency noise that violates physical laws. While empirical investigations confirm bugs in current benchmarks, this cannot theoretically negate the possibility that truncation error could help the model escape spurious correlations in expert data, thereby leading to genuine generalization improvements. Therefore, asserting that the performance improvement from training-free acceleration is "entirely exploiting loopholes" is an over-extrapolation lacking proof of underlying error propagation dynamics.

  2. The improvement plan remains constrained by "hard-coded rules." Although the authors fixed 22 bugs and added process validation to the decider, this is essentially still "patching." As long as benchmark tests rely on rule-based heuristics for successful decisions, all edge cases can never be exhausted. Future new VLA strategies may still find new shortcuts to deceive these upgraded deciders. The paper fails to propose a fundamentally new evaluation paradigm that breaks away from "rule-based decision-making".

  3. Superficial Fixes to the Sim-to-Real Gap: The article points out the distortion of object mass in simulators and corrects it to the real weight for testing, which is very intuitive. However, the Sim-to-Real Gap in rigid body simulators is far more complex, including intricate nonlinear friction, contact dynamics errors, soft body deformation, and sensor noise. Simply modifying the mass is insufficient to fully claim a solution to the limitations of physical assessment. While experiments on real robots (Mobile ALOHA) exist, their depth and scale are insufficient to fully support conclusions generalizing to the real physical world.

Questions for authors
  1. Is the physical mass issue caused by errors in the training data, or is it a problem with the model itself? Would improving the training data prevent the current issue from occurring?

  2. Can you test this on currently popular autoregressive VLA models? The training-free acceleration methods for these models are typically KV-Cache optimization or speculative decoding. Since the error accumulation mechanism of autoregressive models is completely different from that of diffusion models, authors should reproduce this "ranking reversal after bug fix" phenomenon on at least one autoregressive VLA benchmark. If limited to diffusion models, the paper's claim needs to be significantly narrowed.

  3. This time-window-based heuristic remains very fragile. If I design a "cheating strategy"—having the robotic arm move at high speed above the trash can, pause for 250ms, and then forcefully sweep the bottle in—would this new Checker still give a False Success?

  4. Although the paper tested long-horizon benchmarks like VLABench, it only addressed superficial prompt or code logic bugs. For example:Covariate Shift within a bug-free, physically realistic simulation:

  • Extract the per-step truncation error (Δa\Delta a) for the accelerated model.
  • Plot a curve demonstrating how Δa\Delta a accumulates in the State Space at 10, 50, and 200 steps.
  • Explicitly determine the root cause of failure: Does it result from a large single-step Jerk (causing the object to drop), or from the accumulation of micro-errors (causing the gripper to completely miss the target region)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/2):

Part 1/2

We thank the reviewer for recognizing our audit and diagnostic workflow.

W1. Approximation errors and genuine gains

Training-free approximations may improve performance, potentially through implicit regularization. We do not rule this out or attribute all gains to benchmark flaws, and will clarify our wording.

We ask whether checker success signals faithfully reflect task completion. As Q9eB notes, "Treating success-rate gains ... a cheap, reusable audit practice ... validate claims." Anomalous gains motivate inspection but do not establish flaws. Repairs reveal both persistent gains (Table 7: Goal-Language, Object-Language) and gains affected by misjudgment. Place Shoe's unrequested toe-orientation check rejects valid execution and underestimates baseline performance.

Section 4 uses trajectory/checker-region plots, gripper states, instructions, and code to locate and classify defects, then extends the audit to "tasks without anomalous gains." Advice #1 aligns checkers with instructions, also protecting accelerated methods that depart from unnecessary demonstration details but fulfill the task.

Greater action fluctuations do not imply "violating physical laws." Fig. 5 separates policy-output variation from executed motion. We added 20 real-robot trials per method on Pick Dual Bottles; real executed jerk is normalized to baseline:

MethodSim SR (%)Real SR (%)Jerk ratio
Baseline49501.00×
DP-Cache-Fast49251.37×
DP-Cache-Slow46301.35×
ProbeFlow50201.43×

Despite similar simulation SRs, real performance differs. Higher executed jerk accompanied grasping collisions and dropped bottles. Following RoboEval's use of behavioral metrics (Wang et al., 2025, §3.4; Fig. 4), we supplement success with within-scene ranks of policy-output jerk. This follows Appendix D, Advice #4's call to report motion quality and completion time alongside success.

Appendix D ("Gains that ... flaws") explains that persistent gains better support task suitability. Our audit aims to correct misleading comparisons and identify suitable scenarios obscured by flaws.

W2 & Q3. Rule-based fixes and the 250 ms check

Rule-based fixes: Of the 22 bugs, 13 concern initialization and reproducibility, including instruction delivery, initial-state loading, and randomness control. The remaining nine concern instruction–checker–demonstration consistency and are addressed through instruction or checker revisions. Section 4 provides a systematic workflow for identifying and verifying these failures, then extending the audit through shared components and failure mechanisms. The repairs address verified errors, while this reusable diagnostic procedure supports continued examination as new policy behaviors emerge.

250 ms check: The 250 ms interval comes from offline measurements of bottle fall times from positions near the table edge, at heights from tabletop level to 30 cm above it. It is a look-back window: during the 250 ms preceding each bottle’s entry into the acceptance region, the corresponding gripper must have appeared directly above that region. Figure 7(a) shows how this additional evidence distinguishes the observed baseline delivery from DP-Cache-Slow’s knock-and-roll behavior.

Specifically, the proposed counterexample would require the arm to leave the above-bin region, reach the bottle, and sweep it into the bin before that window expires. In the RoboTwin default configuration, physics runs at 250 Hz, allowing approximately 62–63 simulation steps within the window, whereas a policy action typically spans approximately 90 steps after motion processing. These execution timings constrain the proposed sequence. The process check, informed by offline timing measurements, rejects the documented knock-and-roll false successes, demonstrating how our workflow helps identify and diagnose benchmark flaws. We will clarify the timing window and implementation details in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/2):

Part 2/2

W3 & Q1. Mass settings and training data

  1. Mass discrepancies originate in benchmark scene configurations and affect both demonstrations generated under those settings and evaluation dynamics (Fig. 4(c); Table 9). Improving the training data helps policies adapt to corrected physics: In §7.3.2, collecting new demonstrations and fine-tuning under revised Place Fan settings increases baseline SR from 38% to 53% and DP-Cache-Fast SR from 33% to 45%. Appendix A details the training setup.

  2. Our contribution is to demonstrate how physical settings affect acceleration comparisons, rather than to provide a complete sim-to-real solution. We will emphasize in Appendix D that benchmark design should consider less visible physical properties alongside visual realism, using mass as our investigated example and identifying friction and contact dynamics as further factors to examine.

Q2. Autoregressive VLA models

We added experiments with the autoregressive OpenVLA policy and its training-free acceleration method VLA-Cache (KV-cache acceleration), as shown below, SR (%, before→after):

Task / revisionOpenVLAVLA-Cache
Bowl Placement On Stove / BDDL0→860→78
Bowl Placement Next to Plate / BDDL0→400→32
Two-Mug Placement / mass48→3656→22
Place Shoe / checker0→44→4

BDDL fixes (LIBERO-PRO) and checker alignment (RoboTwin) affect autoregressive evaluation. Mass revision in LIBERO Two-Mug changes VLA-Cache's gap from +8 to -14 pp, a point-estimate reversal. These findings extend our audit beyond diffusion without assuming shared error-accumulation mechanisms.

Both failure modes can occur, depending on the task and scene. We provide two cases comparing Baseline and DP-Cache-Fast through execution trajectories, motion metrics, and keyframes to trace how deviations develop into failures (see visualizations).

Accumulating deviations: Select Drink in VLABench. Panel (a) shows the arm trajectories, while panel (b) tracks changes in the object–gripper relative position to characterize grasp drift. Compared with the baseline, the accelerated method exhibits accumulating grasp deviations after action 76, culminating in contact loss at action 127, when the bottle slips from the gripper and the task fails. The baseline maintains its grasp and completes the task. Together, the curves and keyframes illustrate the progression of grasp drift before contact loss.

Abrupt local disturbance: Pick Dual Bottles in RoboTwin. Panel (a) shows the right-arm trajectories, with a close-up of actions 50–60; panel (b) shows the corresponding policy-output action jerk. DP-Cache-Fast exhibits a pronounced jerk peak around action 60, coinciding with local trajectory oscillations and the gripper knocking over the bottle, leading to failure. The baseline changes its actions more smoothly during this phase and keeps the bottle stable. This case illustrates failure associated with an abrupt action disturbance during a critical interaction.

These cases illustrate two distinct failure processes: progressive grasp drift culminating in slippage, and a collision associated with a local jerk peak. Examining state changes, action metrics, and execution keyframes together helps distinguish these processes.

Thank you again for your constructive feedback. We would like to clarify for any further questions.

Reviewer_Q9eBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observed that some training-free VLA acceleration methods (DP-Cache, ProbeFlow) report higher success rates than the baseline policy they approximate, and asks whether such gains reflect benchmark flaws rather than better execution. Using a lightweight workflow that plots object/end-effector trajectories against checker acceptance regions, the authors audit seven simulated manipulation benchmarks (RoboTwin, LIBERO variants, RoboCasa, VLABench, RoboDojo) and report 22 bugs (task consistency, initialization, reproducibility) and 4 design limitations (permissive checkers, fixed thresholds, unrealistic object masses, sim-vs-real action processing). They fix the bugs, tighten checkers, correct masses, and add a motion-aware smoothness score, then re-evaluate Pi 0.5 against three accelerated variants. Results show that method rankings on individual tasks can reverse after correction.

Critical strengths
  • Identifying 22 specific defects across widely used benchmarks and contributing fixes has direct practical value for anyone evaluating VLA policies, independent of the acceleration framing.
  • Treating success-rate gains from training-free approximations as a red flag and localizing causes via trajectory/acceptance-region plots is a cheap, reusable audit practice that could improve how acceleration papers validate claims.
Critical weaknesses
  • Ranking reversals are mostly within noise, and several fixes do not bear on acceleration specifically. Many headline differences are a few percentage points and neither the main text nor the measurement details report the number of evaluation episodes, seeds, or confidence intervals. The "last to first" LIBERO-PRO Spatial-Task result is 0.4 vs 0.6% to 53.2 vs 52.2%: the bug affects all methods equally and the ordering is noise both before and after. Similarly, mass correction drops all methods on Beat Block Hammer, which changes task difficulty rather than exposing an acceleration artifact. Without statistical testing, the central claim that flaws "distort acceleration evaluation" is not established.
  • Narrow and selectively sampled evaluation. Tasks were chosen because they showed anomalous gains, only one base policy (Pi 0.5) and three training-free methods are evaluated, and design-limitation revisions are somewhat ad hoc. The real-robot evidence is limited to jerk statistics on one task. It is therefore unclear how much acceleration gains in the literature are actually explained by these flaws versus how often they are real.
Questions for authors
  1. How many evaluation episodes and seeds underlie each success rate in Tables 2–3? Can you report confidence intervals or paired significance tests for the claimed reversals.
  2. Across all audited tasks (not just those selected for anomalous gains), how often do bug fixes or revisions change the baseline vs accelerated ordering beyond noise? A summary over the full task set would show whether the effect is systematic.
  3. Why should the motion-aware score be rank-based over four methods with zeros for failures? How sensitive are its conclusions to that design, and does action smoothness before post-processing actually predict real-robot success?
2 comments

We thank the reviewer for valuing our fixes and reusable audit workflow. We added statistical analyses and cross-model experiments and clarify our scope below.

W1 & Q1. Evaluation protocol, statistical reliability, and physical settings

We agree that ranking reversals alone have limitations as evidence of distortion in acceleration evaluation. We will avoid ambiguous wording such as "last to first": anomalous gains motivate diagnosis rather than establish a flaw.

Protocol: Each benchmark comprises multiple tasks evaluated over episodes with specific random seeds. Seed selection follows ProbeFlow for LIBERO and the respective published benchmarks for VLABench, RoboTwin, LIBERO-PRO, and RoboCasa.

BenchmarkTasksAverage episodes/seeds per task
LIBERO4050
RoboTwin50100
LIBERO-PRO20050
RoboCasa36550
LIBERO-Plus40250
VLABench60100

Workflow and relative effects. Following Appendix D, Advice #1, we align checkers with instructions and inspect initialization and loaders. After classifying bugs and design limitations, we audit "tasks without anomalous gains," checking each against its specification and implementation.

We define Δ = (SR_accel − SR_base)_before − (SR_accel − SR_base)_after. Positive Δ indicates a smaller acceleration advantage or a larger baseline advantage. Excluding four tasks scored jointly on success and motion quality leaves 28 tasks/subsets for SR analysis, including 10 added through our mechanism-based audit. In 17/28, at least one accelerated method shows a significant paired gap change toward baseline (p<0.05). We will report per-task gap-change estimates and significance-test results in Appendix B.

Physical settings. Section 6.2 studies how unrealistic masses affect acceleration comparisons, rather than how to resolve the sim-to-real gap. In Appendix D we will clarify this scope and recommend documenting and validating mass and inertia.

Appendix D guides assessment of whether acceleration gains reflect valid task execution under matched conditions. Gains that disappear after repair warrant reassessment; persistent gains provide stronger evidence of task-specific benefits.

W2 & Q2. Design revisions and cross-model coverage

Revision criteria. Process checks enforce instructed actions; Dustbin's 250 ms window derives from measured fall times (Appendix B). Mass revisions use real-object references (§6.2), and Scan Object includes a threshold sweep (§6.1). These provide explicit task and physical grounds for the revisions.

We additionally test autoregressive OpenVLA with VLA-Cache on LIBERO-PRO and RoboTwin, and world model FastWAM with DP-Cache and ProbeFlow on RoboTwin. Tables show SR (%, before→after revision).

OpenVLA: the first two tasks fix BDDL instruction delivery in LIBERO-PRO; Two-Mug Placement revises mass in LIBERO; Place Shoe aligns the checker with the instruction in RoboTwin.

TaskBaselineVLA-Cache
Bowl Placement On Stove0→860→78
Bowl Placement Next to Plate0→400→32
Two-Mug Placement48→3656→22
Place Shoe0→44→4

FastWAM (RoboTwin):

TaskBaselineDP-Cache-FastDP-Cache-SlowProbeFlow
Put in Dustbin90→9089→8791→8889→87
Place Shoe93→9793→9492→9293→94
Put in Cabinet92→9288→9287→8784→86
Place Fan94→8384→7294→8093→80

The effects are not uniformly favorable to the baseline: some revisions reduce acceleration gains, some leave the relative gap unchanged, and others improve accelerated variants. This is consistent with our specification-driven audit, which does not use ranking direction as a repair criterion. Together with π0.5, these results show effects across model types and training-free acceleration methods.

Q3. Real-robot evidence and motion-aware scoring

Motion quality and real-robot success: We tested Pick Dual Bottles on Mobile ALOHA, with 20 trials per method. Executed jerk is normalized to baseline:

MethodSim SR (%)Real SR (%)Jerk ratio
Baseline49501.00×
DP-Cache-Fast49251.37×
DP-Cache-Slow46301.35×
ProbeFlow50201.43×

Despite similar sim SR, higher executed jerk accompanies lower real-robot SR.

Why relative ranking: Higher executed jerk accompanied grasping collisions and dropped bottles (Fig. 5), motivating us to assess motion quality alongside success. Because jerk scales vary by task and scene, we rank simulated policy-output jerk within scenes, giving failures zero credit so smooth failures are not rewarded. This combines completion and motion quality, though their optimal combination remains open. RoboEval (Wang et al., 2025) also reports jerk and other behavioral metrics alongside success (§3.4; Fig. 4). Following Appendix D, Advice #4, our score supplements SR.

Sensitivity: The revised appendix will report raw policy-output and executed-motion jerk for alternative joint evaluations of completion and motion quality.

Thank you again for your constructive feedback. We would like to clarify for any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers Q9eB, 2vuK, and cPQR as R1, R2, and R3, respectively.

We appreciate the reviewers’ and the Area Chair’s careful consideration of our work. We appreciate that reviewers recognized the “direct practical value” of our benchmark fixes R1R1, our cross-benchmark audit as a “valuable guide to avoiding common pitfalls” R2R2, and our work as “novel and valuable” R3R3. Our diagnostic workflow was also highlighted as a “cheap, reusable audit practice” R1R1 that can support future benchmark examination R2R2.

Summary of Rebuttal

The initially positive reviewer, R3 (Rating: 3), explicitly confirmed that our explanations “fully addressed my concerns and clarified the scope of the paper’s contributions and conclusions.” R3 maintained their positive rating and increased their review confidence from 2 to 4.

We have provided detailed responses and additional evidence to address R1’s and R2’s concerns. Neither reviewer has raised additional questions to date yet.

Summary of Revisions

We have incorporated additional statistical analyses, OpenVLA and FastWAM evaluations, a Meta-World audit, and real-robot results into the revised manuscript. We have also refined the scope of our claims and expanded the practical guidelines for benchmark design and policy evaluation.

We hope that our responses and revisions address the reviewers’ primary concerns and strengthen the evidence, clarity, and practical usefulness of our work.

We have updated our General Response to summarize the latest reviewer discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.