- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReflectVLA uses paired failed and successful trajectories to guide corrective re-execution. A detector trigger rollback and retrieval of a successful reference. Evaluation focuses on early, reversible failures in LIBERO.
Critical strengths
-
Under shared triggers and budgets, the method recovers 452 episodes, compared with 97 for Retry and 239 for action replay (Table 2). This supports a benefit beyond simply trying again.
-
The paper uses closed-loop rollback rather than restoring simulator states, and includes three-seed results and component ablations. 3.The logic is clear and the method is reasonable.
Critical weaknesses
-
The main evaluation updates the Success Bank and allows two corrections, while the audit uses a fixed bank and the first trigger. Why their gains match exactly is unclear.
-
The detector relies on simulator signals, and training failures are synthetic. It is still unclear how well the method handles natural failures with realistic detection
Questions for authors
-
Can you explain how Tables 1–3 relate and report end-to-end results with a frozen training-only Success Bank?
-
How does recovery change with noisy detection and natural failures, counting rollback within the step budget?
5 comments
Response to Reviewer LKDK Part 2
Response to Weakness 2 & Question 2:
Thank you for raising this important concern. We have extended our evaluation to examine recovery from naturally occurring policy failures under imperfect, non-privileged failure detection, while explicitly accounting for physical rollback within a unified execution budget.
(1) Natural failures versus synthetic training data. We clarify that synthetic failure trajectories are used only for training the correction mechanism. During evaluation, failures arise from the autonomous execution of the original base VLA policies, without injecting OU noise or other artificial action perturbations. Thus, recovery is evaluated on naturally occurring policy failures in simulation, rather than on the synthetic failures used for training. These include unsuccessful grasps, placement deviations, and failed manipulation interactions. We additionally evaluate on LIBERO-Plus, which introduces distribution shifts in camera viewpoints, robot configurations, language instructions, lighting, backgrounds, visual noise, and object layouts.
(2) Recovery under imperfect, non-privileged detection. To address the reliance on privileged simulator signals, we additionally introduce a frozen Qwen3.5-27B-based VLM detector. Unlike the rule-based detector, it uses only causally available RGB observations, robot proprioception, executed actions, and language instructions, without accessing ground-truth object poses, contact states, or simulator task-completion signals.
As reported in Appendix Table 8, the VLM detector achieves 96.84% accuracy, 87.85% recall, and a 2.05% false-positive rate, reflecting imperfect detection with both missed and false triggers. Importantly, ReflectVLA maintains consistent improvements under this detector. On LIBERO (Table 1), the end-to-end success rates with rule-based versus VLM-based detection are:
| Backbone | Rule-based | VLM-based | VLM gain over Retry |
|---|---|---|---|
| OpenVLA | 82.8% | 82.3% | +5.8 pp |
| OpenVLA-OFT | 98.2% | 97.9% | +0.8 pp |
| π0.5 | 98.9% | 98.7% | +2.2 pp |
Across all three backbones and four LIBERO suites, the number of recovered base-policy failures changes from 742/1,940 (38.25%) with rule-based detection to 705/1,940 (36.34%) with VLM-based detection, a decrease of 1.91 percentage points. Since the two detectors can trigger on different failure events, this comparison measures end-to-end recovery rather than conditional recovery on identical triggers.
Furthermore, under VLM-based detection on LIBERO-Plus (Table 2), ReflectVLA achieves 82.0% with OpenVLA-OFT and 89.9% with π0.5, outperforming their corresponding Retry baselines by 9.6 and 9.4 percentage points, respectively. These results provide additional evidence of recovery robustness under imperfect visual observations and distribution shifts.
(3) Physical rollback and unified execution budget. We clarify that all executed simulator control steps count toward the environment-step budget, including normal policy actions, gripper preparation, physical rollback, and corrective re-execution. For each task, all internally evaluated methods and their corresponding baselines use the same maximum environment-step budget, with at most two rollback-and-correction attempts per episode. Rollback is executed through closed-loop tracking of previously visited end-effector poses, without resetting the simulator or restoring object states. Therefore, ReflectVLA receives neither free rollback steps nor additional environment interactions compared with the corresponding Retry baseline.
Overall, these evaluations demonstrate that ReflectVLA retains its recovery advantage on autonomous policy failures under imperfect, non-privileged detection, with physical rollback fully included in the execution budget. We acknowledge that the current evaluation remains simulation-based and focuses on early, recoverable failures. Handling irreversible scene changes and real-world deployment remains future work.
We have clarified these details in Sections 3.5 and 4.2, Tables 1–2, and Appendices A.3–A.4, A.5.2, and A.6.1 of the revised manuscript.
Additional Experiments on Maniskill Plate25 Benchmark
To further strengthen our evaluation, we have added new experiments on the ManiSkill3 Plate25 benchmark (PutOnPlateInScene25Main-v3), as reported in Appendix A.5.3 (Table 7) of the revised manuscript.
Beyond the standard in-distribution setting, we evaluate three out-of-distribution (OOD) categories: visual, semantic, and execution perturbations. We conduct experiments with both OpenVLA-OFT and π0.5 backbones, comparing the original policies, their Retry variants, and ReflectVLA. Both Retry and ReflectVLA use the same VLM-based failure detector and rollback interface to ensure a controlled comparison.
The new results demonstrate consistent improvements:
- OpenVLA-OFT: ReflectVLA achieves an OOD average success rate of 69.78%, compared with 57.15% for Retry (+12.63 percentage points).
- π0.5: ReflectVLA achieves an OOD average success rate of 71.53%, compared with 58.51% for Retry (+13.02 percentage points).
- OOD Execution: The improvements are particularly pronounced under execution perturbations, reaching +18.52 and +20.39 percentage points over the corresponding Retry baselines.
These additional results provide further evidence that the benefits of paired success–failure trajectory conditioning extend beyond the LIBERO benchmark and remain effective under different types of distribution shifts. While real-world deployment remains an important direction for future work, the ManiSkill3 experiments provide an additional independent simulation benchmark supporting the generalization and robustness of ReflectVLA.
We have incorporated these results and the corresponding analysis into Appendix A.5.3, Table 7 of the latest revised manuscript.
Response to Reviewer LKDKPart 1
Response to Weakness 1 & Question 1:
Thank you for highlighting the ambiguity in our evaluation protocol. We have clarified the relationship between the system-level evaluation and shared-trigger audit, and additionally evaluated ReflectVLA with a frozen training-only Success Bank. As the manuscript has undergone revisions, the table numbering has been updated accordingly. All table references below are based on the latest version of the manuscript.
(1) Evaluation protocol and Success Bank usage. Both the main evaluation (Table 1) and the shared-trigger audit (Table 3) allow up to two rollback-and-correction attempts per episode under the same environment-step budget, including physical rollback actions. In the audit, the first trigger serves only as a shared intervention point. Subsequent triggers may still initiate a second correction.
In the main evaluation, each seed/task-specific Success Bank is initialized from training successes and updated with verified successful evaluation trajectories. In the shared-trigger audit, the bank snapshot and retrieved reference are held fixed across comparison conditions at each trigger, rather than freezing the training-only bank throughout evaluation. Thus, the audit controls retrieval differences without changing the underlying online-update protocol.
(2) Relationship between Tables 1, 3, and 8. Table 1 reports end-to-end success rates over 1,500 episodes per LIBERO suite (6,000 in total). Appendix Table 8 provides the rule-based detector statistics, identifying 1,364 true-positive triggers for OpenVLA. Table 3 reports conditional recovery on these same triggers.
Importantly, the Full and Retry results in Tables 1 and 3 are derived from the same matched episode-level evaluation records, rather than independent evaluations. Specifically, Full ReflectVLA recovers 452/1,364 triggered failures, while Retry recovers 97/1,364. Their per-suite differences in recovered episodes are 96, 71, 113, and 75, corresponding to end-to-end success-rate gains of 6.4, 4.7, 7.5, and 5.0 percentage points, respectively, after normalization by 1,500 episodes per suite. Overall, the 355 additional recovered episodes correspond to 355/6,000 = 5.92 percentage points, matching the 5.9-point gain reported in Table 1 for OpenVLA with the rule-based detector.
Therefore, the exact correspondence arises from the same underlying recovery outcomes reported with different denominators, rather than independently obtained identical improvements. Table 3 uses true-positive triggers as the denominator for conditional recovery, whereas Table 1 uses all evaluation episodes for end-to-end success.
(3) End-to-end evaluation with a frozen training-only Success Bank. We additionally evaluated ReflectVLA with online bank updates disabled, retaining only successful training demonstrations throughout evaluation. As reported in the revised Table 4, the frozen-bank variant achieves the following end-to-end success rates under the same single-seed setting:
The frozen-bank variant achieves 82.45%, only 0.45 percentage points below the online-updated model under the same setting. This indicates that the observed recovery improvements do not substantially depend on online Success Bank updates.
We have clarified these protocols and their relationship in Sections 3.5, 4.2, and 4.3, and Appendices A.2, A.4, and A.7. The frozen-bank results are included in Table 4 of the revised manuscript.