acceptodds
Under review as a conference paper at ICLR 2027

How Many Samples Should a Planner Draw? The Answer Depends on a Convention Nobody Reports

Abstract

Sampling-based planners (CEM, MPPI) are the deployment-time engine of modern world-model agents, and a practitioner's first question about them is how to spend planner compute: draw more candidate action sequences, or refine them over more iterations? We show that the first half of this question is not well posed as usually asked. Implementations differ in whether the elite set is a fixed fraction of the population (mbrl-lib's default, used by PETS and PlaNet) or a fixed count (TD-MPC2's convention), and this choice, usually inherited from a library default, can reverse the sign of the measured effect of drawing more samples. On identical PlaNet models, seeds and episodes, increasing the population changes return by under one convention and under the other (both ); the other two PlaNet tasks point the same way, while on PETS the convention changes the effect's size (significantly only on Ant) but not its sign. Instrumenting the optimizer gives a post-hoc mechanism: the elite count sets how concentrated the refit proposal is, so tying it to the population widens the proposal as sampling grows (PETS dispersion ), while a fixed count tightens it (). Sample-count comparisons are therefore not commensurable unless the elite convention is stated. The second half of the question holds the population fixed, so the convention does not enter: across released TD-MPC2 agents on 14 tasks in two domains and two further agent families, refinement iterations are the more reliable way to convert planner compute into return. They recover full-planner performance on weak-prior high-dimensional bodies and on pixel PlaNet, where width does not (e.g. , , where the best width-only configuration still falls short), with 90% of the gain arriving between the second and fourth iteration on all four TD-MPC2 tasks swept (4-D to 38-D); on state-based PETS one well-configured wide pass matches them, and on a poorly fit model they hurt. How much refinement a task needs tracks the prior-to-task gap, a contrast carried by three weak-prior high-dimensional bodies. Every sweep was preregistered with dated amendments; of the twenty predictions tested, four held, twelve failed, one held in part and three could not be established, and we report all of them.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, sampling-based planning, cross-entropy method, MPPI, world models, reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZjdGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This empirical study investigates how sampling width, iterative refinement, and policy priors affect model-based planning. Its central finding is that the effect of increasing population size depends on whether the number or the fraction of retained elites is held fixed. Experiments with TD-MPC2, PETS, PlaNet, and MyoSuite examine this convention, the value of repeated distribution updates, and the role of the policy prior. The paper also records predictions that failed and proposes a post-hoc explanation based on proposal concentration.

Critical strengths
  1. The elite convention is a consequential experimental variable, and the paper tests it directly. The comparisons in Section 8.1 and Table 2 show that a population-size sweep can simultaneously change selection pressure when elites scale with the population. The interaction tests survive Holm correction in the three PlaNet settings and PETS Ant, and the measured width effect changes sign across all three PlaNet settings. The paper appropriately distinguishes these interactions from significance of each individual slope. This is useful new knowledge for interpreting planner-scaling results, independent of whether a new algorithm is introduced.

  2. The decomposition of planning into prior quality, sampling, and refinement makes the study informative beyond the main convention effect. Sections 3–5 and 8 examine these components across different agent families and tasks. Table 3 retains failed predictions, and Section 9 distinguishes the observed effects from stronger explanations that the data do not support. In particular, the paper identifies P4 as post hoc and rejects a universal contraction rate. This makes the negative results scientifically useful and helps establish the limits of the proposed account.

Critical weaknesses

W1. The general concentration claim in P4 needs stronger assumptions. Section 7 argues that fixed-count elites concentrate as population size grows, while fixed-ratio dispersion approaches its limit from below. IID candidates and additive score noise do not establish these statements about dispersion in action space. For example, with a constant underlying objective and independent score noise, selection is independent of the actions, so a fixed number of selected actions retains the proposal distribution regardless of population size. Figure 5 supports the mechanism in the measured settings, but does not establish it under the stated assumptions. This limits the general explanation of the convention effect, rather than invalidating the controlled return measurements.

W2. The cross-family compute-allocation recommendation goes beyond the available budget comparisons. Sections 8.1 and 10 use the width-versus-iteration results to favor repeated refinement. However, the PETS and PlaNet endpoint comparisons in Table 6 use different total rollout budgets: PETS compares up to 1,400 single-iteration candidates with 350×8=2,800350\times8=2{,}800 across iterations, and PlaNet compares 4,000 with 1,000×10=10,0001{,}000\times10=10{,}000. Some TD-MPC2 configurations permit comparisons at similar total rollout counts, but that does not establish the cross-family recommendation. The results show that refinement helps under the tested settings, and its advantage as a use of a fixed compute budget needs a more direct comparison.

Questions for authors

Q1 (W1). What assumptions on the objective, proposal, and scoring noise are needed for P4's action-space concentration claim? Can the statement be restricted to those conditions, with Figure 5 presented as evidence for the measured settings rather than for the current general claim?

Q2 (W2). For PETS and PlaNet, does the preference for additional refinement remain when width and iteration count are compared at the same total number of model rollouts? If sequential computation changes the practical recommendation, how does that comparison look in wall-clock time?

2 comments
AuthorsOpenReview ↗

Equal-rollout comparisons added (refinement wins on PlaNet and TD-MPC2, not PETS); P4 restated with the assumptions it needs

Thank you. Both points are right, and the revised PDF (changes in blue) narrows both claims. The results below come from the per-episode records of the submitted sweeps, except for the simulation under Q1, which we added after submission. No agent was re-run.

Q2 / W2. Equal rollout budgets. The submitted sweeps contain matched rollout budgets on PlaNet and TD-MPC2. They are now reported in Section 8.1 ("Width against refinement at equal model rollouts").

  • PlaNet, 100 candidates × 10 iterations (1,000 rollouts) against 1,000 × 1 with a fixed elite count: refinement is higher on all three tasks, on 10 of 10 paired episodes each (p = 0.002). The returns are 484.9 vs 260.5 on cheetah run, 195.6 vs 104.0 on walker walk and 522.7 vs 111.0 on finger spin. The walker walk margin (+92) is below the 116-point nondeterminism floor in Table 6, so we read that task for direction only.
  • TD-MPC2, released checkpoints, 512 × 4 against 2,048 × 1 with 4 fixed elites (the better width setting): 542.0 vs 524.6 on hopper-hop (p = 0.008 over 30 paired episodes), and 833.1 vs 327.4 on dog-walk (higher on 3 of 3 seeds; those two runs do not share evaluation episodes, so we give no p-value).
  • PETS has no exact pair. In the nearest one, 350 × 2 (700 rollouts) against 1,400 × 1 with a fixed count, refinement is ahead only on Hopper and not significantly (+43, p = 0.20), width is better on Walker2d (−68, p = 0.029), and on Ant the direction depends on the elite convention.

So at equal rollouts the recommendation holds on PlaNet and TD-MPC2 and does not hold on PETS. Rule (1) in Section 10 and the abstract now say exactly that.

On wall-clock: these sweeps did not record time. Iterations are sequential while width parallelises on a GPU, so a latency budget can favour width. Section 10 now states the rule per model rollout, not per unit of latency.

Q1 / W1. Conditions for P4. Your counterexample is correct. With a constant objective and independent noise, selection ignores the action, and a fixed count keeps the proposal's dispersion at every N. P4 in Section 7 and Appendix C now adds two assumptions:

(a) the objective is not constant and has a unique maximiser in the proposal's support; (b) the score noise is independent of the action, with tails no heavier than Gaussian.

(b) is needed too: with exponential-tailed noise, conditioning on an extreme score tilts the proposal by a fixed amount, and concentration stops.

To check the statement we replaced the left panel of Figure 5 with a documented simulation (the panel shows the Gaussian case; all four cases are in Appendix C): a 6-D Gaussian proposal, a quadratic objective, 200 repeats, N from 50 to 22,400, and 10 fixed elites against a 10% ratio. The fixed-count dispersion behaves as follows:

  • without noise it falls 0.630 → 0.213;
  • with Gaussian noise of s.d. 1 it falls 0.653 → 0.386;
  • with Gumbel noise of s.d. 3 it levels off, 0.752 → 0.710 (Gaussian noise of the same s.d. still falls, 0.733 → 0.550);
  • with a constant objective (your case) it stays at 0.967 → 0.958.

We also deleted "indeed broader, approaching the asymptote from below". With a constant objective the fixed-ratio setting also rises (0.92 → 1.0), so that rise is at least partly the downward bias of a standard deviation estimated from few elites. Figure 5 is now presented as evidence for the measured settings only.

We correct one more point about Figure 5 ourselves: the PETS panel is transcribed from the probe's recorded summary table (12 replans per cell, three decimals), and per-replan values were not kept. The caption now says so.

Reviewer_ZjdGOpenReview ↗

Response to Authors

Thanks for the clarifications and the added analyses. The new assumptions and simulations address my concern about P4. The equal-rollout comparisons in Section 8.1 also answer W2, and the revised claims now reflect the PETS results and the lack of timing measurements.

What I am less convinced by now is how much new insight the paper offers as a whole. The clearest contribution is showing that changing the elite convention can reverse the effect of drawing more samples. The other findings seem less new. Iterative CEM and elite selection were already studied in work such as PlaNet. The new comparisons show that refinement helps in the tested settings, but tell us less about when to expect it to work better than wider sampling. Section 4 also leaves the effects of prior quality and action dimensionality difficult to separate.

I am moving my recommendation to 3 (Weak acceptance). The controlled finding on the elite convention is useful enough to support acceptance, though I still find the contribution beyond that result limited.

Reviewer_EjjXOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies sampling-based planners for world-model agents. The paper points out that implementation differ based on whether the elite set is a fixed fraction of the population or a fixed count. The paper shows that this difference can dramatically affect, and even reverse the sign of, the effect of drawing more samples. Second, the paper shows that how many refinement iterations a task needs tracks the prior-to-task gap.

Critical strengths

Good points but not that surprising. (I may be wrong about this. This is not my area.)

Experiments were preregistered. The authors are honest that only four of the twenty hypotheses held.

Critical weaknesses

Not that surprising. (I may be wrong about this. This is not my area.)

No theoretical results.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

No new theory, agreed; the sign reversal, and what the revision changes

Thank you. We agree that the paper has no new theory; Appendix C says that none of its mathematics is new. The paper's contribution is a set of controlled measurements. In the revision (changes in blue) the one mechanism statement, P4 in Section 7, is restated under the narrower assumptions it needs, as Reviewer ZjdG asked.

The result we think is not expected is a sign reversal. On the same PlaNet models, evaluation seeds and episodes, drawing 40× more samples changes return by −63 when the elite set is a fixed fraction of the population and by +246 when it is a fixed count (both p = 0.002). The revision adds a related case on a released TD-MPC2 agent (hopper-hop, Section 8.1). There, widening from 32 to 2,048 samples turns a flat effect under a fixed count (+19, not significant) into a large loss under a fraction (−372), and the two effects differ with p < 0.001 over 30 paired episodes. So whether "more samples" helps cannot be read from a result unless the elite rule is stated. Implementations differ in this rule, and among six published sweeps of the number of samples that we found, four do not say which rule they used.

Other changes:

  • the title now reads "...Depends on an Unstated Convention";
  • the iteration result is described as confirming existing defaults, with comparisons at an equal number of model rollouts;
  • the preregistration table is no longer listed as a contribution.
Reviewer_EjjXOpenReview ↗

Read the rebuttal

I read the rebuttal and would like to keep my score unchanged.

Reviewer_YdZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between sequential iterative refinement and wider search when planning with a previously trained world model and using it for planning to achieve tasks. They find that the elite-set convention can reverse whether more samples improve performance. More refinement iterations can recover planner performance on high-dimensional tasks where the prior is weak.

  1. The elite-set convention (fixed fraction vs. fixed count) can reverse whether more samples help.
  2. Refinement iterations, not wider sampling, recover full planner performance, mainly on high-dimensional, weak-prior tasks.
  3. How much planning a task needs tracks how poor the policy prior is, with action dimensionality as a proxy.
  4. Most of their 20 preregistered predictions failed.
Critical strengths
  • Pre-registered predictions are good experimental procedure.
  • The elite-convention confound is a useful practical warning for anyone comparing planners across codebases.
  • They use publicly released checkpoints, so results are verifiable, and their full-planner arm reproduces the released performance, which validates the harness.
  • Covers 35 tasks, which increases the reliability that these findings generalize more broadly
Critical weaknesses
  • Limited model diversity behind the key claims. All TD-MPC2 results come from a single checkpoint release, and the PETS/PlaNet agents were trained by the authors on three tasks each. The elite-convention result, the most novel finding, rests on six cells from two families, as the paper itself acknowledges.
  • The preregistration ledger adds limited value as a contribution. Many predictions are highly specific (e.g., exact saturation points, orderings across tasks), so a high failure rate is unsurprising given evaluation noise, and the failures themselves are not very informative.
  • Unclear what prior belief is overturned. The recommendation (use 4–6 iterations) matches existing defaults: TD-MPC2 uses 6, PETS 5, PlaNet 10. The paper doesn't show anyone believed width was the better lever.

Notes

  • The characterisation of Hamrick et al. (2021) as studying "discrete domains" is imprecise: their study included control tasks (with discretized actions), and they found exceptions where test-time search mattered.
Questions for authors
  1. Can you point to published work whose conclusions would change under the other elite convention?
1 comment
AuthorsOpenReview ↗

Two convention cells on released TD-MPC2 agents; what we found on Q1; the iteration result now framed as confirming defaults

Thank you. We agree with most of this and have narrowed the paper accordingly; the changes are in blue in the revised PDF. All returns and tests below come from the per-episode records of the submitted sweeps; no agent was re-run.

W1. Breadth behind the convention result. We accept the point: TD-MPC2 is one release with three seeds, and we trained the PETS and PlaNet agents ourselves. The one addition we can make from the submitted record is this. The elite-dilution control in Section 5 applies the same manipulation to the released TD-MPC2 checkpoints: max(4, N/8) is a fixed 1/8 fraction above N = 32, and the control holds 4 fixed elites. Widening from 32 to 2,048 samples at one iteration gives:

  • hopper-hop: −372 under the fraction (p < 0.001), against a flat +19 under the count (p = 0.23). The two effects differ with p < 0.001 over 30 paired episodes, in the same direction on each of the 3 seeds.
  • dog-walk: +77 against +257 (both p < 0.001), larger under the count on 3 of 3 seeds.

The submitted paper reported these cells as elite dilution, not as convention cells. Section 8.1 now reports them as a convention contrast and labels this as a post hoc regrouping. The limitations in Section 10 say the same. The result now rests on six cells from two families plus two cells on released TD-MPC2 checkpoints that we did not train. This adds a family, but not a release.

Q1. Published conclusions under the other convention. We cannot point to a published conclusion that we have shown to flip, and we do not claim one. What we could document from public code and papers is this. The original PETS and PlaNet code keep a fixed elite count (PlaNet: 100 of 1,000 candidates). MBRL-Lib's implementations of the same agents keep a fraction (10% for PlaNet cheetah run), and our PETS and PlaNet agents use MBRL-Lib. On the same PlaNet model, the two rules give opposite answers to "do more samples help?" (Table 2), so a width sweep run on the original code and one run on the library would be expected to disagree on these tasks. Among six published CEM/MPPI papers we found that sweep the number of samples, two state the elite rule in force (both a fixed count), and four do not say which rule held during the sweep. So the divergence is documented in implementations, not in a published reversal. Accordingly the abstract and Section 2 now attribute the fraction rule to MBRL-Lib rather than to PETS and PlaNet, and the title now reads "...Depends on an Unstated Convention".

W3. Which belief is overturned. You are right that 4–6 iterations is already the default, and the paper should not call it a surprise. The introduction now says that the iteration result confirms the defaults (TD-MPC2 6, PETS 5, PlaNet 10). It also says where they matter: on PlaNet and on weak-prior TD-MPC2 bodies, width does not substitute for refinement even at an equal number of model rollouts. On PlaNet, 100 × 10 beats 1,000 × 1 on all three tasks (p = 0.002 each; on walker walk the margin is below the noise floor of Table 6); on TD-MPC2, 512 × 4 beats 2,048 × 1. On PETS it does not, so rule (1) in Section 10 is narrowed. What is not a default is that the measured sign of "more samples" depends on a setting that implementations choose differently and most of the published sweeps we found do not state.

W2. The preregistration ledger. Agreed that it is a reporting device rather than a contribution. It is no longer in the contributions list, and Table 3 stays as the full record. Two of its failures did shape the paper, and neither was a near-miss. Row 2 failed with the opposite sign (ρ = +0.55, p = 0.009) and became the dimensionality result. Row 20 reversed, which is why P4 is labelled post hoc.

Hamrick et al. (2021). Thank you; Section 2 now describes their study as MuZero across discrete-action domains, including control tasks with discretised actions, and notes the exceptions where search at test time matters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.