- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A failed robot trajectory is not automatically a useful training example. Under a fixed correction budget, the key question is which logged states deserve new expert actions. This paper studies the offline acquisition problem for vision-language-action policies and introduces EDGE (Experience Discovery via Gap-aware Evaluation). A nominally trained, frozen latent world-action ensemble supplies three ranking signals: action-consequence surprise, a teachability score that measures the local mass of progress-improving action alternatives, and a confidence weight that discounts uncertain residual transport. The observed transition residual anchors local counterfactuals without refitting the nominal models to the target pool.
Critical strengths
This paper studies which failed states should receive expert corrections under a fixed correction budget for post-training vision-language-action models. The problem is important, and the experimental design has some controlled elements.
Critical weaknesses
The current evidence is not strong enough to support a robust advantage over strong baselines, and the method’s novelty and generalization remain limited.
-
The statistical evidence for the main comparison is not strong enough. The main experiments use only five random seeds, so statistical power is limited. On the PhysicalShift benchmark, although EDGE outperforms G-bandpass in all five paired runs, the advantage is not significant after multiple-comparison correction, and the smallest possible two-sided result is still around 0.06. In other words, the current advantage may be driven by seed variation. The authors should add more independent seeds, tasks, or stratified resampling, and they should fix the main comparator in advance rather than choosing the strongest baseline after seeing the results.
-
Method novelty and theoretical guarantees are limited. EDGE mainly combines residual anchoring, distance decay, ensemble uncertainty, and a progress verifier. Similar ideas already exist in prior work. The authors also admit that the confidence weight is not a strict guarantee, that it cannot handle shared bias, and that the local smoothness assumption fails under action delay. The paper needs a clearer explanation of its irreplaceable contribution relative to prior work, and it needs either clearer theoretical conditions or stronger negative controls.
-
Hyperparameter tuning and comparison fairness are not transparent enough. The center of G-bandpass is tuned on development tasks and then frozen, but it is not fully clear whether EDGE’s several key hyperparameters are determined on the same development set, with the same budget and the same amount of tuning. If EDGE uses more validation information or more tuning opportunities, the main table advantage may be overestimated. In addition, matching budget by the number of action chunks ignores different action horizons and expert time costs, and cross-backbone comparisons also need more caution.
-
Generalization and deployment value remain limited. All experiments are concentrated in the same simulator family, and there is no real-robot or cross-simulator validation. The PhysicalShift benchmark mainly uses hand-designed single-factor perturbations, and the set of compound conditions is small. Kernel decay does not help under action delay, which suggests that the so-called support edge may depend heavily on smooth dynamics assumptions. Therefore, the current conclusions should not be directly generalized into a universal acquisition ru
Questions for authors
Q1. The statistical evidence for the main comparison is not strong enough. A1. We agree that five seeds limit statistical power, and we will not overstate significance. The key evidence is a consistent paired advantage: on PhysicalShift, EDGE is about 3.0 percentage points above the frozen G-bandpass in all five paired seeds, with a 95% t interval of roughly 1.6 to 4.4 points. Under the conservative 24-comparison Bonferroni adjustment, the result is not significant at the 0.05 level, and the exact sign-flip sensitivity is also limited by the small number of seeds. In revision, we will (i) pre-specify G-bandpass as the primary comparator, (ii) report effect sizes and seed-level results more prominently, (iii) add seeds if compute permits, and (iv) soften claims that require a multiplicity-robust win on both benchmarks.
Q2. Method novelty and theoretical guarantees are limited. A2. We agree that the individual components are not new. Our contribution is the formulation and controlled study of corrective-label acquisition under matched supervision and policy-update budgets, together with the same-snapshot rollback test that isolates residual transport. We will clarify the distinction from DAgger-style gating, ThriftyDAgger, DataMIL, and CheckVLA. We will also state the theoretical limits more directly: the local bound is conditional, the confidence weight is a heuristic rather than a certificate, and kernel transport fails under action delay. The negative controls in Appendix Q already illustrate this; we will make them more prominent in the main text.
Q3. Hyperparameter tuning and comparison fairness are not transparent enough. A3. All selectors share the same pool, query budget, temporal exclusion window, replay mixture, and update schedule. The G-bandpass center is tuned on disjoint development tasks and frozen before validation. EDGE hyperparameters are fixed from nominal calibration and are not refit to the target pool. In revision, we will add a table listing tuning data, search ranges, and fixed values for EDGE and all baselines. We will also clarify that matching by action-chunk count is a within-backbone budget and does not equate expert time across different action horizons.
Q4. Generalization and deployment value remain limited. A4. We acknowledge that all main results are within one simulator family. We will soften universal claims and add cross-simulator and real-robot validation as explicit future work. We already include SmolVLA, leave-one-factor-out acquisition, compound shifts, independent pools, and imperfect correction channels as robustness checks; we will present them as evidence of scope rather than as proof of broad generalization.