- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes the SufficientPlan framework, which improves the computational efficiency of WM-based planning by selecting a sufficient planning budget using PSBC and reducing redundant encoding operations using SCR.
Critical strengths
A notable advantage of the framework is that it can be applied to existing world models without requiring retraining. The experimental results across a variety of models and tasks show meaningful reductions in computational cost while preserving task success rates.
Critical weaknesses
-
While the framework effectively reduces planning costs at deployment time, the calibration process itself may introduce additional computational overhead. I would be interested to know whether the overall approach remains efficient once the calibration cost is taken into account, particularly in settings with a relatively small number of deployments. In addition, the framework evaluates candidate budgets sequentially using fixed increments. Could the authors clarify the motivation behind this design choice? A comparison of calibration costs, as well as a discussion of alternative search strategies, would help clarify the overall efficiency of the proposed method.
-
The framework assigns a fixed planning budget to each model–task pair. However, planning difficulty may vary considerably across states within the same task. For instance, different object configurations or environmental conditions may require more extensive search. In such cases, a fixed budget could lead to unnecessary computation in easier states, or to failures in more challenging ones. Could the authors comment on the robustness of the method under such variation? An analysis of state-dependent planning difficulty, or a comparison with adaptive planning methods, would help address this concern.
Questions for authors
The main questions addressed in the Critical Weaknesses above.
7 comments
I appreciate the authors' detailed responses and additional experiments. However, my main concern regarding the use of fixed planning budgets remains insufficiently addressed.
While the additional distance-based analysis provides some evidence of robustness across different initial configurations, it does not fully capture state-dependent planning difficulty. In particular, without a comparison with adaptive planning methods, it remains unclear whether the proposed fixed-budget approach provides a reasonable trade-off between computational efficiency and task performance across different states.
Therefore, I increase my score. However, as my main concern has not been fully resolved, my assessment still leans toward rejection.
Thank you for acknowledging our additional analyses and for raising your score. To directly address your remaining concern, we have added a comparison with online state-adaptive planning in Section C.9, “Comparison with State-Adaptive Planning,” of the newly revised manuscript.
We evaluate Full, PSBC, and three compute configurations of an online adaptive baseline on 500 matched held-out episodes per method on INTACT/Push-T. The adaptive baseline monitors improvement in elite candidate costs and stops each planning solve accordingly, allowing search depth to vary across states. Its thresholds are selected on development episodes and frozen before testing. All methods use the same pretrained model, candidate population, planning horizon, and evaluation episodes, with SCR disabled.
PSBC achieves the same observed success rate as Full and the high-compute adaptive configuration (93.0%), while reducing planning time by 72.9% and 69.1%, respectively. Compared with the medium-compute adaptive configuration, PSBC uses 28.2% less planning time and achieves a 1.4-percentage-point higher observed success rate. More aggressive adaptive stopping reduces computation further, but success decreases to 90.0%.
This comparison directly supports the practical value of a calibrated fixed budget in the tested setting. Although planning requirements vary across states, online adaptation does not automatically produce a better computation–performance trade-off: PSBC retains Full’s observed success rate at substantially lower cost, while the lowest-cost adaptive configuration trades some task success for further savings.
The newly added table is reproduced below. SR intervals are Wilson 95% confidence intervals. Time is warmed, GPU-synchronized, batch-amortized planning time on RTX A5000 GPUs, including adaptive stopping overhead. Adaptive denotes our implemented online elite-cost-stall baseline.
| Method | Held-out SR, % [95% CI] | Mean candidate evaluations/episode | Mean planning time (s/episode) | Iterations/solve: mean [P10, P90] |
|---|---|---|---|---|
| Full | 93.00 [90.42, 94.92] | 9,882.0 | 1.420 | 30.00 [30, 30] |
| PSBC (2,400) | 93.00 [90.42, 94.92] | 2,683.2 | 0.384 | 8.00 [8, 8] |
| Adaptive (low compute) | 90.00 [87.06, 92.33] | 1,366.2 | 0.200 | 3.90 [3, 7] |
| Adaptive (medium compute) | 91.60 [88.84, 93.73] | 3,735.0 | 0.536 | 11.24 [3, 22] |
| Adaptive (high compute) | 93.00 [90.42, 94.92] | 8,711.4 | 1.246 | 26.69 [16, 30] |
As the rebuttal period draws to a close, we hope that our responses have effectively addressed your concerns. Thank you again for your time and consideration.