- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Backtrack, a recovery module that enables a frozen VLA to recover from failures during online execution. Backtrack consists of a failure detection module and a recovery module. The failure detection module uses the VLA’s internal features to classify whether the current episode will succeed or fail, and the recovery module reverses previously executed motion to return the robot to a state similar to an earlier one (backtracking). The method is evaluated with three VLA backbones (SmolVLA, π0.5, and π0-FAST) on two benchmarks (LIBERO and MetaWorld), and it improves success rates in most settings, with the largest gains on the long-horizon LIBERO-Long suite.The paper addresses an important but challenging problem. Failures in robot manipulation are diverse and can arise in many different forms, which makes recovery difficult to address, yet important. The proposed module is lightweight and requires neither recovery demonstrations nor fine-tuning of the VLA. Since only a small head is trained, the training cost is low and the approach is efficient. The method is simple, yet it includes the components needed for detection and recovery.
Critical strengths
- The paper addresses an important but challenging problem. Failures in robot manipulation are diverse and can arise in many different forms, which makes recovery difficult to address, yet important.
- The proposed module is lightweight and requires neither recovery demonstrations nor fine-tuning of the VLA. Since only a small head is trained, the training cost is low and the approach is efficient. The method is simple, yet it includes the components needed for detection and recovery.
Critical weaknesses
-
Weak baselines. The paper states that retrying after a detected failure is not new (line 136). Given this, the paper should explain why it does not compare against existing approaches such as Bellman-Guided Retrials (Du et al., 2024) or LLM-based failure recovery methods (Lin et al., 2025; Liu et al., 2023b). The current comparisons are limited to the unassisted policy, a longer step budget (Appendix A.3), and privileged restoration (Appendix A.6).
-
Recovery from irreversible failures. The introduction motivates the method with cases such as dropping or knocking over an object. In such cases, reversing the arm’s motion may cause the arm to contact the object again. In addition, backtracking is performed only at the kinematic level, and the scene is not restored. Section 3.2 (around line 223) and Section 6 acknowledge these points and state that the experiments do not isolate the sources of improvement. However, the paper does not explain why returning only the arm configuration is sufficient for recovery. A more detailed rationale, or an analysis of which failure types are recovered, would strengthen the paper.
-
Insufficient qualitative results. The paper does not include videos or snapshot sequences of the recovery process. Without them, it is difficult to see what kinds of failures are recovered and how, which weakens the paper’s claims.
-
Sensitivity to the threshold. The threshold sweeps (Tables 5–7 and 14) suggest that performance depends on the threshold and on its interaction with the retreat rule. On π0.5, the adaptive rule at t = 0.90 performs significantly worse than control. On π0-FAST, the ranking of thresholds changes with the retreat shape. The selected configurations also differ across policies (t = 0.97, 0.99, and 0.90). The paper does not discuss how the threshold should be chosen for a new policy or task without a per-policy sweep.
Questions for authors
-
Training the classifier requires a substantial amount of rollout data including failures (7,953–20,000 episodes per policy, Table 2). Is it reasonable to assume that this amount of outcome-labeled rollout data is available, particularly on a physical robot where each rollout must be labeled as success or failure? In addition, the failure prediction head is trained on rollouts from the same four LIBERO suites used for evaluation, so its generalization to unseen tasks is not evaluated.
-
Would it be possible to add comparisons with existing failure recovery methods?
-
Could the authors explain in more details with figures and examples of why backtracking at the kinematic level alone is sufficient for recovery in many cases?
-
Could the authors provide qualitative results, such as videos or snapshot sequences of the recovery process?
-
Could the authors provide guidance on selecting the threshold and retreat rule for a new policy or task, given the sensitivity observed in Tables 5–7?
Response to Reviewer oVMC
We thank the reviewer for the detailed and constructive review, and for recognising that Backtrack addresses an important problem with a lightweight module that needs neither recovery demonstrations nor fine-tuning of the VLA.
W1/Q2 (baselines).
LLM-based recovery methods (Lin et al., 2025; Liu et al., 2023b) add a second large model at inference, which Backtrack is designed to avoid. Bellman-Guided Retrials (Du et al., 2024) is the closest in spirit, but its authors note that it "assumes access to a robust recovery policy, which may not be trivial for more involved tasks" (their Sec. 6); Backtrack replaces this assumed recovery policy with the arm's own recorded motion. We could not reimplement these methods within the discussion period.
W2/Q3 (recovery from irreversible failures).
Execution errors such as a missed or slipped grasp leave the scene largely unchanged (see the new figure under W3); returning the arm to where it was before the error is then enough for a fresh attempt. Dropping or knocking over an object changes the scene, but the task can remain solvable from there, so a fresh attempt can still succeed. We agree that an analysis of which failure types are recovered would strengthen the paper and leave it to future work.
W3/Q4 (qualitative results).
We agree. In the revision, we will add a figure showing one full LIBERO-Long episode as a snapshot sequence (missed grasp → retreat → re-grasp → success) together with the predicted failure probability, plus further sequences, including failed recoveries, in the appendix.
W4/Q5 (sensitivity to the threshold).
A strict threshold is a safe default: t = 0.99 improves over the control on all three policies under each policy's reported retreat rule, with every interval excluding zero (Tables 5–7). Loose thresholds are riskier on a policy that rarely fails, likely because they trigger needless retreats (π0.5, adaptive t = 0.90). For SmolVLA, settings also transfer across benchmarks: in a new experiment, its LIBERO-Long settings, run unchanged on MetaWorld, raise success from 72.32% to 76.64% (+4.32 pp, seed-paired 95% CI [+1.35, +7.29]), and every threshold from 0.90 to 0.995 gives +3.6 to +4.3 pp. We will add a recommendation: start at t = 0.99, and tune only if a sweep is affordable.
Q1 (amount of rollout data).
The head needs only rollouts of the unmodified policy labelled by episode success, with no teleoperation or per-step labels. The amounts in Table 2 are what we used; how few rollouts suffice, and thus how practical collection is on a physical robot, we have not tested and leave to future work.
Regarding unseen tasks, the MetaWorld head is trained only on the 11 medium-difficulty tasks (l. 354–355). On the 39 tasks it never saw, Backtrack with the transferred LIBERO settings still improves success by +2.41 pp [+0.09, +4.73], compared with +4.32 pp over all 50 tasks; we will add this breakdown to the revision. On LIBERO, generalisation to unseen tasks remains untested, and we will state this as a limitation.