- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RESETTLE, an add-on recovery layer for frozen VLA/world-action policies. It samples two action chunks under identical conditioning and flags trouble when their disagreement exceeds a training-data P95 threshold at two consecutive decisions. It then retrieves the nearest same-task demonstration frame with a V-JEPA-2-AC encoder fine-tuned on demonstrations. One corrective action is built from a state-servo prior plus a guarded visual residual, and control returns to the base policy. Evaluation covers six base policies on LIBERO-Plus, Meta-World, and RoboCasa Tabletop, plus LIBERO-Pro Swap with a GPT-5.5 planner and four real-world tasks with two policies. Reported gains range from about 2 to 9 points. The paper also reports 74–93% lower monitor-plus-recovery computation than VoLoAgent, and lower than Agentic Robot and an adapted LPB.
Critical strengths
-
The method is simple, needs no base-policy fine-tuning, and works across many policies and benchmarks, including a real robot and a planner-based system. Consistent gains of a few points across this breadth suggest a practical plug-in, which is what pushes me toward acceptance.
-
The analysis is candid and useful. Disagreement separates failed from successful trajectories much better than V-JEPA feature distance. The paper also reports how rarely recovery actually takes over and includes explicit failure cases. This makes the claims easier to trust and interpret.
Critical weaknesses
-
The statistical evidence is thin: Meta-World uses 10 episodes per task, LIBERO-Pro Swap 100 trials, and the real-world tests 20 trials per task, with no seeds, confidence intervals, or significance tests, so several gains (e.g., +2.09 on LDA) could be noise. The recovery module is also trained on benchmark demonstrations that the frozen baseline never sees, and there is no control such as random-trigger recovery or a small fine-tuned corrector to isolate the benefit of the trigger and retrieval.
-
The latency comparison is not like-for-like: competing methods are re-implemented or adapted (LPB), produce different outputs (a trajectory versus one action), and totals exclude nominal inference and execution, with no end-to-end or equal-wall-clock success comparison. The "lightweight" framing also sits uneasily with a ~1B-parameter encoder plus nearest-neighbor search over the full demonstration bank. Disagreement further misses the consistent-but-wrong failures the paper itself shows.
Questions for authors
- Can you provide confidence intervals or multiple evaluation seeds, especially for Meta-World, LDA/RoboCasa, and LIBERO-Pro Swap? Which gains survive?
- Can you add controls that isolate the benefit of the trigger from the benefit of the corrective action, e.g., triggering at random with the same intervention frequency, or always-on recovery?
- How does a fine-tuned or DAgger-style baseline trained on the same demonstrations compare? The recovery module sees demonstration data the baseline does not.
- What are the memory and retrieval costs as the reference bank grows (e.g., for Meta-World's 204k frames), and how does retrieval time scale?
- Does the P95 threshold generalize when the test distribution differs sharply from training? Is any per-benchmark tuning of thresholds or guard parameters done?
2 comments
Thank you again for your time and thoughtful feedback. With two days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.
Thank you for recognizing the simplicity, broad applicability, and candid failure analysis of our method. We address the concerns about statistical reliability, trigger selection, and computational cost below.
1. Multiple seeds and statistical reliability
We completed evaluations with five different random seeds for both the baseline and RESETTLE. The table reports mean success rates (%) ± sample standard deviation across the five runs.
Meta-World uses the same averaging convention over the four difficulty groups as the paper. The reported uncertainty is standard deviation. All six comparisons show positive gains in all five runs. Notably, the LDA/RoboCasa improvement remains consistent, with a mean gain of 2.8 percentage points and individual-run gains of 2.1–4.3 points. These additional results support the repeatability of the improvements across evaluation seeds.
2. Separating trigger selection from corrective actions
We added random triggering on LIBERO Plus with QwenPI, keeping retrieval and recovery fixed. H denotes the predicted action-chunk length, and K denotes the number of actions executed before re-observation and replanning during nominal execution.
Under H8/K4, random recovery improves success over no recovery by 1.3 percentage points, while our trigger provides a further 1.5-point improvement at nearly identical intervention frequency. This supports the additional value of selecting when to intervene. Under H16/K8, our trigger also achieves higher success.
3. Demonstration access
We clarify that our recovery module uses demonstrations from the same training dataset used to train the base policy; it receives no additional demonstrations. Both recovery training and reference-library construction use existing training data, without additional expert corrections or DAgger-style data collection. The distinction is therefore how the demonstrations are used, rather than access to additional demonstration data.
4. Reference-library memory and retrieval costs
Default retrieval searches only same-task candidates, rather than the entire library. Measurements on LIBERO Plus are:
For Meta-World, the library contains 204,806 frames, averaging 4,180 same-task candidates. Features and states occupy 1,103.16 MiB, the CPU memory increase after loading is 1.08 GiB, and median retrieval latency is 2.54 ms. Memory grows approximately linearly with total frame count. Search latency increases with same-task candidate count, with a steeper increase for larger LIBERO Plus libraries.
We also agree that “lightweight” characterization should refer to measured online computation and sparse intervention, rather than a small encoder parameter count.
5. Threshold generalization and selection
Our reported thresholds are calibrated at P95 using offline inference on training data, without tuning on test outcomes. For each base policy on LIBERO Plus, one threshold is shared across all variants, tasks, and suites. Appendix Table 12 reports the following QwenPI sensitivity analysis:
Different variants favor different thresholds: Robot performs best at P85, whereas Noise performs best at P99. We do not select the best threshold per variant; all main results use the training-derived P95 setting. All four percentiles improve overall success, supporting robustness to threshold choice under the evaluated shifts.
Finally, we acknowledge that consistently incorrect behavior can evade disagreement detection. Our supplementary analysis found that 146 of 1,859 failed QwenPI trajectories received no alarm throughout execution out of 10,030 trajectories overall. This represents 7.85% of failures, indicating a relatively low overall missed-alarm proportion.