- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PREEMPT, an inference-time framework that refines the actions of a frozen VLA policy. It has four components:
- A V-JEPA + GRU anticipator (PTA) predicts “physical decision windows” to decide when to search.
- A dual-view latent world model predicts 5-step consequences of candidate actions. The Vendi Score of these predicted trajectories schedules the interval before the next search.
- A completion-time evaluator, trained on completed, failed, and right-censored attempts, maps short-horizon predictions to the probability and timing of operation completion.
- A 12-D spline correction in the policy’s intent coordinates is optimized with CEM, using structural costs that penalize deformation and drift across replans.
Critical strengths
Strengths
- The motivation is well grounded. The observation that the value of correction depends on the interaction stage is intuitive, and concentrating search around critical windows is a clean idea.
- The completion-time evaluator is a novel design. Modeling discretized completion-time distributions with a partial-label likelihood handles completed, failed, and censored samples in a principled way, and is better motivated than direct progress regression.
- The intent-constrained correction balances smoothness with fidelity to the policy’s motion structure. The cumulative-offset cost addresses consistency across replans, which similar work often overlooks.
Critical weaknesses
Weaknesses
- Small simulation gains with unclear significance. On LIBERO, PREEMPT adds only 0.4 points for π0.5 and 0.2 points for OpenVLA-OFT, and it degrades OpenVLA-OFT on the Object suite. The appendix mentions paired bootstrap, but the main text reports no confidence intervals, so it is unclear whether the gains are significant.
- No component ablations. The paper does not isolate the contribution of each module. Useful comparisons would include replacing the completion-time evaluator with a simple progress or success classifier, removing the structural costs, replacing spline CEM with sampling from the policy itself, and removing Vendi-based scheduling. Without these, the necessity of the core design choices is unsupported.
- Problematic RoboCasa baseline. The H=5 baseline itself degrades sharply (from 454 to 319 successes), so the reported 42.3% failure reduction overstates the effect. Against the default H=50 baseline, the gain is only 26 successes, and Double Door performance decreases.
- Narrow scope of the window-guided computation result. Refinement is enabled on only 3 LIBERO-10 tasks (150 episodes), yet results are aggregated over all 10 tasks. Both the 58.4% reduction and the 95% vs. 92% success comparison are therefore easy to misread. Stronger baselines, such as fixed-interval or random triggering, are also missing.
Questions for authors
Questions
- How well calibrated is the completion-time evaluator? Do its predicted Q values match empirical completion rates?
- How often is a correction actually executed (J > 0), and how many executed corrections prevent a failure?
- How much training data do PTA, the world model, and the evaluator require on the real robot? What is the annotation cost for transferring to new tasks?
3 comments
Response to Reviewer DPUU (2/3)
W3: RoboCasa execution horizon and continuity
Thank you for asking us to clarify the source of the RoboCasa improvement and the drop in baseline success when H changes from 50 to 5. We explain the purpose of the H=5 comparison and the policy's sensitivity to replanning below:
-
Native policy configuration. We use the publicly released LeRobot π0.5 RoboCasa checkpoint, whose prediction and default execution lengths are both 50. With H=5, the policy still predicts 50 actions but discards the remaining 45 after executing the first five. This increases replanning frequency tenfold and can disrupt temporal consistency across independently generated chunks.
-
Fairness of comparison. PREEMPT executes local corrections over five steps. We also evaluate π0.5 with five-step replanning to examine whether more frequent feedback alone can explain the improvement. This baseline matches the local correction granularity.
-
RoboCasa task characteristics. Door manipulation requires sustained contact along a constrained hinge trajectory. Interruptions from frequent replanning may help explain the decline in H=5 success, particularly on door tasks in the original Table 2.
We evaluated H=5 + continuity correction to separate continuity effects from predictive action selection:
| Method | Success |
|---|---|
| π0.5, H=50 | 454/700 |
| π0.5, H=5 | 319/700 |
| π0.5, H=5 + continuity correction | 409/700 |
| PREEMPT (full) | 480/700 |
PREEMPT retains a 10.14-percentage-point advantage over the continuity baseline. Together with the structural-cost ablations in W2, this supports a contribution beyond smoothing execution across replans.
W4: Search allocation and stronger scheduling baselines
Our original allocation study covered a limited set of tasks. To strengthen the scheduling evidence, we compared Every replan, fixed-interval scheduling, random scheduling, and fixed-interval scheduling within PDWs. Each search evaluates 4 × 64 = 256 sampled candidates, and the fixed interval is 20 steps:
| Search schedule | Success (%) | Candidate evaluations per episode |
|---|---|---|
| Every replan | 80.8 | 14,748.16 |
| Fixed-interval scheduling | 86.2 | 3,660.80 |
| Random scheduling | 87.2 | 5,767.68 |
| Fixed-interval scheduling within PDW | 88.2 | 1,886.72 |
| PREEMPT (full) | 93.6 | 3,706.88 |
PREEMPT uses 74.9% fewer candidate evaluations than every replan and outperforms fixed-interval scheduling by 7.4 percentage points at nearly the same search cost. Restricting fixed-interval scheduling to PDWs achieves higher success with approximately half the search cost of unrestricted fixed-interval scheduling.
Q1: Calibration of the completion-time evaluator
We thank you for asking the completion-time evaluator's estimation reliability. Q combines the probability and timing of local operation completion. As defined in Section 4.3, it is the expected discounted completion utility: The completion-time bin upper bounds are 0, 5, 15, 30, 60, 120, and 240 control steps; non-completion has zero utility. Accordingly, we compare completion probabilities with observed completion frequencies, and Q with observed discounted utilities using the same time-bin weights.
We conducted the evaluator evaluation on 208 RoboCasa parent trajectories, disjoint from evaluator and target-domain WM training/model selection, as well as score-scale calibration. Probability metrics use 2,646 windows with resolved outcomes, and utility MSE uses 1,990 windows with resolved utility targets. We further compared predictions and observations in ten equal-width prediction bins. In the most populated probability bin, 0.9–1.0, the mean prediction is 0.9384, closely matching the observed completion frequency of 0.9453 across 676 windows. In the most populated Q bin, 0.2–0.3, predicted Q averages 0.2512, versus an observed discounted utility of 0.2639 across 586 windows.
| Evaluator | Completion Brier ↓ | Completion ECE ↓ | Discounted-utility MSE ↓ |
|---|---|---|---|
| PREEMPT | 0.1899 | 0.1067 | 0.0890 |
These results show particularly close agreement at high predicted completion probabilities. Lower-probability predictions exhibit greater underestimation, while the high-probability region provides encouraging evidence of reliable completion estimates.
(2/3)
Response to Reviewer DPUU (3/3)
Q2: Frequency and benefit of executed corrections
We report both the frequency of executed corrections and their effect on task outcomes. The table below gives paired outcomes for the original LIBERO and RoboCasa results in Tables 1 and 2, using 2,000 and 700 episodes per policy comparison, respectively. The confidence intervals describe the success-rate difference in percentage points. A rescue changes a baseline failure into success; a harm changes a baseline success into failure:
| Refined policy | Baseline | Rescues | Harms | 95% CI (pp) |
|---|---|---|---|---|
| LIBERO | ||||
| π0.5 + PREEMPT | π0.5 | 10 | 2 | [+0.06, +0.74] |
| OpenVLA-OFT + PREEMPT | OpenVLA-OFT | 10 | 6 | [-0.19, +0.59] |
| OpenVLA-OFT + PREEMPT | OpenVLA-OFT, H=5 | 38 | 8 | [+0.84, +2.16] |
| RoboCasa | ||||
| π0.5 + PREEMPT | π0.5, H=5 | 181 | 20 | [+19.41, +26.59] |
| π0.5 + PREEMPT | π0.5, H=50 | 44 | 18 | [+1.53, +5.90] |
We also measured correction frequency and compared individual interventions from the same states on LIBERO-10. One continuation executes the nominal five-step prefix and the other executes the correction; both then follow the same π0.5 policy. Correction frequency is the percentage of replanning decisions at which a correction is actually executed.
| Method comparison | State pairs | Rescues | Harms | Both succeed | Both fail | Correction frequency (%) |
|---|---|---|---|---|---|---|
| PREEMPT vs. π0.5 | 500 | 24 | 14 | 444 | 18 | 36.3 |
Results show that 24 corrections change failure to success, while 14 change success to failure in this branch study. These counts measure the outcome of the selected intervention directly.
Q3: Real-robot training data and annotation cost
The three auxiliary models used in the reported real-robot evaluation share 565 demonstrations, split as shown below.
| Model | Training trajectories | Validation trajectories | Test trajectories |
|---|---|---|---|
| PTA | 444 | 58 | 63 |
| World model | 444 | 58 | 63 |
| Completion-time evaluator | 444 | 58 | 63 |
Annotation cost for new tasks: We annotate physical decision windows in approximately 141 trajectories per task. Human annotation, verification, and correction require approximately 0.783 person-hours per task. The world model learns from recorded actions and observations and requires no additional manual labels.
Concluding Remarks
Thank you for recognizing the motivation and completion-time formulation novelty, and for suggesting more direct tests of the components and baselines. We have provided:
- Component and continuity comparisons: the completion-time evaluator gains 11.0 points over either simpler scorer, and PREEMPT exceeds continuity correction by 10.14 points.
- Stronger scheduling controls: PREEMPT achieves 93.6% success and uses 74.9% fewer candidate evaluations than Every replan.
- LIBERO uncertainty and broader evaluation: we report paired confidence intervals and achieve 88.6% on LIBERO-Plus, up from 85.2%.
- Calibration and deployment details: we quantify evaluator calibration, correction frequency and outcomes, and real-robot training data and annotation cost.
We sincerely appreciate your constructive feedback. We are glad to hear of your willingness to reconsider the rating and look forward to further discussion.
Reference
[1] Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., Fu, J., Gong, J., & Qiu, X. (2025). LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv preprint arXiv:2510.13626.
(3/3)
Response to Reviewer DPUU (1/3)
We appreciate your recognition of the motivation for physical decision windows, the completion-time formulation, and the role of intent-constrained correction. Thank you for the constructive suggestions that prompted the component comparisons and additional analyses below.
W1: LIBERO gains and statistical uncertainty
We report paired outcomes and confidence intervals to quantify the LIBERO gains. We use LIBERO to evaluate general manipulation, where π0.5 and OpenVLA-OFT already achieve 96.7% and 96.5% success, leaving limited room for improvement. PREEMPT raises these rates to 97.1% and 96.7%, respectively. The table below reports results for 2,000 episodes per policy comparison. The confidence intervals describe the success-rate difference in percentage points. A rescue changes a baseline failure into success; a harm changes a baseline success into failure.
We also evaluated PREEMPT on LIBERO-Plus [1], where π0.5 has more room for improvement. Average success rises from 85.2% to 88.6%, and PREEMPT exceeds the reported OpenVLA-OFT result by 17.2 percentage points. These results show a larger benefit under the variations tested by LIBERO-Plus.
W2: Contributions of the individual components
We conducted the requested component ablations on RoboCasa. Each tested component contributes to the full result: success falls from 68.6% to 60.7% without the cumulative-offset cost, to 58.0% without all structural costs and to 60.3% without both predicted futures and Vendi adaptation. The completion-time evaluator improves success by 11.0 percentage points over either the binary success classifier or the progress predictor. Replacing spline CEM with policy sampling, using 256 candidates per search, reduces success by 17.6 points. The scheduling comparison in W4 tests PTA and Vendi adaptation: full PREEMPT achieves 93.6%, compared with 88.2% for fixed-interval scheduling within PDWs, which omits Vendi adaptation.
(1/3)