- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This empirical study investigates how sampling width, iterative refinement, and policy priors affect model-based planning. Its central finding is that the effect of increasing population size depends on whether the number or the fraction of retained elites is held fixed. Experiments with TD-MPC2, PETS, PlaNet, and MyoSuite examine this convention, the value of repeated distribution updates, and the role of the policy prior. The paper also records predictions that failed and proposes a post-hoc explanation based on proposal concentration.
Critical strengths
-
The elite convention is a consequential experimental variable, and the paper tests it directly. The comparisons in Section 8.1 and Table 2 show that a population-size sweep can simultaneously change selection pressure when elites scale with the population. The interaction tests survive Holm correction in the three PlaNet settings and PETS Ant, and the measured width effect changes sign across all three PlaNet settings. The paper appropriately distinguishes these interactions from significance of each individual slope. This is useful new knowledge for interpreting planner-scaling results, independent of whether a new algorithm is introduced.
-
The decomposition of planning into prior quality, sampling, and refinement makes the study informative beyond the main convention effect. Sections 3–5 and 8 examine these components across different agent families and tasks. Table 3 retains failed predictions, and Section 9 distinguishes the observed effects from stronger explanations that the data do not support. In particular, the paper identifies P4 as post hoc and rejects a universal contraction rate. This makes the negative results scientifically useful and helps establish the limits of the proposed account.
Critical weaknesses
W1. The general concentration claim in P4 needs stronger assumptions. Section 7 argues that fixed-count elites concentrate as population size grows, while fixed-ratio dispersion approaches its limit from below. IID candidates and additive score noise do not establish these statements about dispersion in action space. For example, with a constant underlying objective and independent score noise, selection is independent of the actions, so a fixed number of selected actions retains the proposal distribution regardless of population size. Figure 5 supports the mechanism in the measured settings, but does not establish it under the stated assumptions. This limits the general explanation of the convention effect, rather than invalidating the controlled return measurements.
W2. The cross-family compute-allocation recommendation goes beyond the available budget comparisons. Sections 8.1 and 10 use the width-versus-iteration results to favor repeated refinement. However, the PETS and PlaNet endpoint comparisons in Table 6 use different total rollout budgets: PETS compares up to 1,400 single-iteration candidates with across iterations, and PlaNet compares 4,000 with . Some TD-MPC2 configurations permit comparisons at similar total rollout counts, but that does not establish the cross-family recommendation. The results show that refinement helps under the tested settings, and its advantage as a use of a fixed compute budget needs a more direct comparison.
Questions for authors
Q1 (W1). What assumptions on the objective, proposal, and scoring noise are needed for P4's action-space concentration claim? Can the statement be restricted to those conditions, with Figure 5 presented as evidence for the measured settings rather than for the current general claim?
Q2 (W2). For PETS and PlaNet, does the preference for additional refinement remain when width and iteration count are compared at the same total number of model rollouts? If sequential computation changes the practical recommendation, how does that comparison look in wall-clock time?
2 comments
Response to Authors
Thanks for the clarifications and the added analyses. The new assumptions and simulations address my concern about P4. The equal-rollout comparisons in Section 8.1 also answer W2, and the revised claims now reflect the PETS results and the lack of timing measurements.
What I am less convinced by now is how much new insight the paper offers as a whole. The clearest contribution is showing that changing the elite convention can reverse the effect of drawing more samples. The other findings seem less new. Iterative CEM and elite selection were already studied in work such as PlaNet. The new comparisons show that refinement helps in the tested settings, but tell us less about when to expect it to work better than wider sampling. Section 4 also leaves the effects of prior quality and action dimensionality difficult to separate.
I am moving my recommendation to 3 (Weak acceptance). The controlled finding on the elite convention is useful enough to support acceptance, though I still find the contribution beyond that result limited.
Equal-rollout comparisons added (refinement wins on PlaNet and TD-MPC2, not PETS); P4 restated with the assumptions it needs
Thank you. Both points are right, and the revised PDF (changes in blue) narrows both claims. The results below come from the per-episode records of the submitted sweeps, except for the simulation under Q1, which we added after submission. No agent was re-run.
Q2 / W2. Equal rollout budgets. The submitted sweeps contain matched rollout budgets on PlaNet and TD-MPC2. They are now reported in Section 8.1 ("Width against refinement at equal model rollouts").
So at equal rollouts the recommendation holds on PlaNet and TD-MPC2 and does not hold on PETS. Rule (1) in Section 10 and the abstract now say exactly that.
On wall-clock: these sweeps did not record time. Iterations are sequential while width parallelises on a GPU, so a latency budget can favour width. Section 10 now states the rule per model rollout, not per unit of latency.
Q1 / W1. Conditions for P4. Your counterexample is correct. With a constant objective and independent noise, selection ignores the action, and a fixed count keeps the proposal's dispersion at every N. P4 in Section 7 and Appendix C now adds two assumptions:
(a) the objective is not constant and has a unique maximiser in the proposal's support; (b) the score noise is independent of the action, with tails no heavier than Gaussian.
(b) is needed too: with exponential-tailed noise, conditioning on an extreme score tilts the proposal by a fixed amount, and concentration stops.
To check the statement we replaced the left panel of Figure 5 with a documented simulation (the panel shows the Gaussian case; all four cases are in Appendix C): a 6-D Gaussian proposal, a quadratic objective, 200 repeats, N from 50 to 22,400, and 10 fixed elites against a 10% ratio. The fixed-count dispersion behaves as follows:
We also deleted "indeed broader, approaching the asymptote from below". With a constant objective the fixed-ratio setting also rises (0.92 → 1.0), so that rise is at least partly the downward bias of a standard deviation estimated from few elites. Figure 5 is now presented as evidence for the measured settings only.
We correct one more point about Figure 5 ourselves: the PETS panel is transcribed from the probe's recorded summary table (12 replans per cell, three decimals), and per-replan values were not kept. The caption now says so.