- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how prediction errors in world models influence the selection of candidate action sequences by sampling-based planners. Through error analysis, the authors derive sufficient conditions for "elite-set" decision-making, which ensuring that the optimal action sequence remains within the chosen subset, and propose EliteCert. This approach uses a primary model to filter candidates, followed by a secondary, independent verification model to re-score those candidates that could potentially alter the composition of the elite set. Experimental results demonstrate that this method improves closed-loop control performance while requiring less additional computation compared to evaluating the entire candidate set.
Critical strengths
-
The research question is valuable. By linking world model errors to the candidate selection process of a planner, the study offers a clear perspective on why the returns from increasing the search budget eventually saturate.
-
The methodology is sound, and the experiments are robust. The selective re-ranking design is intuitive; the experimental evaluation includes closed-loop validation, comparisons of computational overhead, and rigorous baselines using single-model approaches. Furthermore, the authors document instances where the verifier fails or offline performance gains fail to generalize, which strengthens the credibility of their conclusions.
Critical weaknesses
-
The attribution of certain performance gains remains insufficiently explained. For MPPI methods that rely on the magnitude of cost values, normalization can alter the weights assigned to candidate trajectories; therefore, it is necessary to further disentangle the effects of ranking adjustments from those resulting from changes in cost scaling.
-
The scope of applicability remains limited. Evaluations have focused on simulation tasks within a single world-model framework, and the observed gains depend on the availability of a suitable verifier. Furthermore, the reliability of candidate filtering and bandwidth calibration under distribution shifts remains unclear.
Questions for authors
-
For MPPI, is it possible to disentangle the performance gains, attributable to re-ranking versus those resulting from changes in weight scaling, by using a control baseline normalized by the same cost criteria?
-
When should the validator qualification test and calibration bandwidth be re-evaluated following shifts in task distribution or changes to the model? What are the applicable boundaries for these methods, based on existing results?
2 comments
Thank you to the authors for the late response. Your response have addressed most of my concerns.
The description in response 2 doesn't seem to directly prove that a sufficiently high band width coverage is linked to a validator being qualified, does it?
I choose to keep my intial score.
Thank you for the insightful review. Please also see Point 1 of our General Comment, which contextualizes the headline Push-T numbers against stronger single-model baselines.
W1 / Q1. Separating elite re-ranking from cost scaling in MPPI. We ran the exact closed-loop control experiment you suggested (Section 6.1, Appendix E.2). In MPPI, returned costs c~i=sdB(c)si affect both which K candidates are selected and the softmax weight spread (sdB(s)<1 when models disagree). We evaluated two controlled arms on the identical 600 Push-T tasks, 3 planner seeds (1,800 paired runs/arm), temperature, band, and model calls:
W2. Empirical scope. We agree that our evaluation is scoped to LeWM latent world models, simulated benchmarks, and settings where an error-diverse validator exists (stated explicitly in Section 8). Within this scope, two new results clarify the picture: (i) against the strongest single model, ensemble gains are +2.3 pp for CEM (150x10), +7.9 pp for MPPI, and -0.4 pp for iCEM; and (ii) at the default 300x30 budget—where native search has saturated—EliteCert reaches 95.4%, adding +2.2 [+0.9, +3.6] pp over the best single checkpoint at 300x30 (Table 2).
Q2. When must qualification and band width δ be re-evaluated, and what are the boundaries? Section 6.3 and Appendix E.1 now establish concrete empirical boundaries. Both checks should be re-evaluated when the planning model, validator set, task distribution, or search budget (N,K,T) changes: