acceptodds
Under review as a conference paper at ICLR 2027

Not All Evaluations Are Equal: Where Test-Time Compute Matters in World-Model Planning

Abstract

World-model planners spend their test-time compute scoring populations of imagined action sequences, and they spend it uniformly across the iterations of the search. We find that the planning decision is made early. In cross-entropy-method (CEM) planning with pretrained latent world models, late-round evaluations can be approximated or removed without a detectable change in success on PushT, Reacher, and Two-Room, whereas approximating early rounds can derail the search. This is consistent with the contraction of the proposal: as it narrows, a linear response operator fitted at run time around the proposal mean halves its held-out error on PushT and cuts it more than fourfold on Reacher. Substituting this operator into the early rounds lowers success significantly on PushT under CEM, iCEM, and MPPI and on Two-Room under MPPI, and raises the final cost on both tasks, whereas score noise of the same cost scale shows no such stage dependence on any of the three tasks. Reacher marks the boundary: its success is flat from 5 to 30 rounds, and early substitution does not hurt and even raises success. Whether early approximation hurts goes together with an offline quantity, the cost of stopping the native search early: the two agreed in all eight settings of a test with predictions fixed in advance. The finding can be used in two ways that trade time against fidelity: stopping two rounds early plans 12% faster at a small but significant increase in final cost, and Exact-to-Local CEM, which keeps early rounds exact and approximates late ones, has the same success outcome as native CEM on all 200 paired PushT decisions with 13% fewer exact queries, 3% less time, and no detectable increase in final cost. For the planners and world models we tested, late-round compute can be cut first; the early rounds are where approximation is risky.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, model-based planning, test-time compute, cross-entropy method, model predictive control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FrP4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the importance of exact world-model evaluations changes during population-based planning. The authors fit a local linear response model from a small number of exact queries and use it to replace evaluations at different search stages. Early replacement is generally more damaging than late replacement on PushT and Two-Room, whereas Reacher does not show the same pattern. The analysis relates these results to proposal contraction. On PushT, representation error decreases during search while candidate-ranking agreement changes little. The authors then introduce Exact-to-Local CEM, which retains exact evaluations early, approximates intermediate rounds, and evaluates the final population exactly. Experiments compare this schedule with native CEM, smaller populations, truncated search, and adaptive stopping, with additional stage interventions in iCEM and MPPI. The main finding is that late computation can be reduced without a detectable loss of success in the tested settings, with different trade-offs in planning time and final model cost.

Critical strengths
  1. The paper shows that the effect of approximate evaluations depends on when they are introduced. Table 1 shows larger losses from early than late replacement in several PushT and Two-Room settings. On PushT, decreasing representation error is not accompanied by a substantial improvement in ranking agreement (Table 6), yet late replacement can preserve success. This suggests that the effect on the search, beyond approximation accuracy itself, matters when allocating exact evaluations. Reacher provides a boundary case, since early replacement does not reduce success there.

  2. The local model in Section 4 replaces world-model evaluations while retaining the original cost function and CEM update rule. The operator is fitted from the current decision’s own queries and requires no retraining of the world model. This makes it useful both for studying the effects of replacement and for implementing E2L. Appendix H supports the design through elite-overlap and proposal-drift measurements, and the timing experiments show that the resulting planner can reduce complete planning time.

Critical weaknesses

My main reservation concerns the additional practical benefit of E2L over stopping earlier. In Table 2, E12 saves about 3% of planning time on PushT, whereas R13 saves about 12% with similar observed success rates. E12 has a smaller reported mean increase in final model cost relative to full CEM (+0.19 versus +0.69), but it also takes longer. The comparison therefore establishes a trade-off without resolving whether approximate refinement makes better use of the available planning time. This leaves the analysis of search stages as the stronger contribution.

Questions for authors

In Table 2, E12 and R13 use nearly the same number of exact queries, but E12 takes 0.152 s per decision compared with 0.138 s for R13. How does E2L compare with truncated native CEM in final model cost when their planning times are similar? A comparison on the same evaluation sources would clarify whether the approximate refinement rounds offer a benefit at a fixed planning-time budget.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FrP4

Thank you for the accurate summary. You point out that the effect of an approximate evaluation depends on when it enters the search, which is the central observation of our paper. As you also note from Table 6 of the submission, where exact evaluations are allocated, an approximation's effect on the search matters beyond its accuracy. The revision tests the practical consequence of this finding: whether, at comparable planning time, cheap local refinement trades cost against time better than truncated native CEM.

1. Additional practical benefit of E2L over early stopping.

Yes. At the middle budget on PushT, E2L-CEM uses 12.5% less planning time with no detectable difference in pre-success cost, −0.01 with 95% interval −0.25 to 0.22. At the low budget on Two-Room, it lowers pre-success cost by 6.37, interval 5.31 to 7.45, with 4.7% less planning time. At the high budget, its cost is below native CEM with 45 rounds, 9.31 against 9.78 on PushT and 42.06 against 43.54 on Two-Room, with 1.4% and 5.4% more planning time, and its observed mean is close to the 9.28 and 42.14 of native CEM with 60 rounds, with 24% and 7% less planning time. Section 6.4 now reads:

"At the high budget, E2L-CEM lowers pre-success cost against native CEM with 45 rounds by 0.47 [0.30, 0.64] on PushT and 1.49 [0.83, 2.13] on Two-Room, using 1.4% and 5.4% more planning time. Its observed success and mean pre-success cost are close to those of native CEM with 60 rounds (PushT 3,126 against 3,115 successes, cost 9.31 against 9.28; Two-Room 2,483 against 2,483, 42.06 against 42.14), in 0.436 against 0.577 s and 0.560 against 0.604 s per decision."

Development runs selected the same schedule, E24/8, independently on both tasks: the first 24 of 60 rounds are exact and the local operator is refitted every 8 rounds after that. So at budgets where truncation already preserves success, the added local updates recover most of the model cost that truncation loses.

2. Final model cost of E2L against truncated native CEM at comparable planning time on the same sources.

At comparable planning time on the same sources, E2L-CEM has lower pre-success cost at three of the six budgets, both high budgets and the low budget on Two-Room. On PushT at the low and middle budgets no difference was detected, and at the middle budget on Two-Room its cost is 0.81 higher with 12.2% less time. Each time ratio is measured with both planners on the same GPU, at the low and high budgets over 25 disjoint timing sources (Table 2 footnote, Appendix C). Two-Room uses 2,950 sources with N=150, and PushT 3,375 sources with N=300.

TaskBudgetE2L vs nativeSuccesses E2L / native, McNemar pPre-success cost difference, 95% intervalTime ratio
Two-RoomlowE16/8 R=40 vs R202,415 / 2,430, 0.13−6.37, −7.45 to −5.310.953
Two-RoommiddleE24/8 R=60 vs R362,461 / 2,466, 0.64+0.81, 0.12 to 1.510.878
Two-RoomhighE40/8 R=100 vs R452,483 / 2,477, 0.53−1.49, −2.13 to −0.831.054
PushTlowE14/8 R=36 vs R203,086 / 3,096, 0.25+0.06, −0.23 to 0.370.965
PushTmiddleE24/8 R=60 vs R353,104 / 3,112, 0.28−0.01, −0.25 to 0.220.875
PushThighE36/8 R=90 vs R453,126 / 3,119, 0.31−0.47, −0.64 to −0.301.014

Within each budget no success difference is significant. Across budgets, lower cost goes with more successes: high-budget E2L-CEM solves 3,126 PushT and 2,483 Two-Room episodes against 3,096 and 2,430 for native CEM with 20 rounds (p=1.4×10⁻³ and 6.9×10⁻⁶).

The E12 against R13 comparison you mention matched exact queries. E12 takes 0.152 s per decision and R13 takes 0.138 s, because planning time also includes fitting and local updates. The new experiments therefore choose comparison points by complete planning time and test whether added local refinement improves the cost and time trade-off, and the original comparison remains in Appendix E. So at comparable planning times, E2L-CEM significantly lowers pre-success cost relative to the evaluated truncated-CEM references at both high-budget points and at the low-budget Two-Room point.

Thank you for the detailed response. It addresses most of my concerns. However, after rereading the paper, I have one further question.

The paper mainly reports absolute changes in Cost but does not provide the baseline Cost values, making it difficult to assess the magnitude of these changes. Could the authors also report the baseline Cost and the corresponding percentage changes? I also noticed that in some experiments, Cost changes substantially while the success rate remains nearly unchanged. Could this be due to the relatively loose success criteria of these tasks? For tasks requiring higher precision, could similar changes in Cost lead to a substantial drop in success rate? Would E2L still maintain its advantage in such settings?

Sorry for bringing this up a bit late. If you still have time, I'd appreciate your thoughts on these questions.

Thank you for rereading the paper and for this further question. We are glad our response addressed most of your concerns, and the question is not late at all.

1. Baseline cost and percentage change

New Appendix S gives the native cost, written below as mean (median), and each change in the same form. A source's cost averages the exact final cost over its decisions.

Section 6.4 and Table 2, pre-success cost at comparable planning time:

TaskComparisonNativeE2LChangeE2L lower on
PushTE14/8 vs R2012.20 (2.83)12.26 (2.29)+0.5% (−19.3%)76%
PushTE24/8 vs R3510.22 (1.66)10.21 (1.50)−0.1% (−9.4%)78%
PushTE36/8 vs R459.78 (1.41)9.31 (1.24)−4.8% (−12.3%)96%
Two-RoomE16/8 vs R2061.21 (10.45)54.84 (2.52)−10.4% (−75.9%)79%
Two-RoomE24/8 vs R3645.77 (2.03)46.58 (1.15)+1.8% (−43.2%)72%
Two-RoomE40/8 vs R4543.54 (1.28)42.06 (0.79)−3.4% (−38.5%)93%

The last column is the share of sources where E2L-CEM costs less. The mean is set by sources neither arm solves: on low-budget PushT, 254 such sources average 106, while on the 3,061 sources both arms solve, E2L-CEM lowers the median cost from 2.59 to 2.10. Native CEM with 60 rounds reaches 9.28 (1.25) on PushT and 42.14 (0.92) on Two-Room, which high-budget E2L-CEM matches.

Section 6.2 and Table 1, cost change from stage substitution:

Task and plannerNativeEarlyLate
PushT CEM R=608.84 (1.24)+185.0%+6.4%
Two-Room CEM R=6020.63 (0.51)+230.9%+1.5%
PushT CEM14.60 (3.83)+37.7%+4.9%
PushT iCEM71.04 (35.06)+78.4%+11.5%
PushT MPPI16.60 (5.70)+178.8%+9.0%
PushT CEM K=1514.83 (3.35)+27.4%+12.0%
PushT CEM K=6019.59 (6.62)+16.3%+3.3%
PushT LeWM11.01 (3.00)+46.8%+11.1%
Two-Room CEM18.38 (0.86)+91.1%+10.1%
Two-Room MPPI8.36 (0.43)+970.4%−31.2%
Reacher MPPI23.06 (0.44)+98.4%−1.1%
Reacher CEM35.37 (0.56)+20.7%−32.0%

In Section 6.3, stopping native CEM after 13, 11, 7 and 3 of 15 rounds raises the PushT mean of 20.84 (4.22) by 0.3%, 15.6%, 57.6% and 141.5%, and stopping after 15 of 30 rounds raises the Two-Room mean of 14.93 (0.79) by 53.5%. On Section 6.2's 2,000 sources, early noise and early local scoring raise the first-decision mean of 12.63 (2.82) by 23.7% and 38.4%.

So early substitution raises cost by 16% to 970%, and late substitution by at most 12% where its interval excludes zero. At comparable planning time, E2L-CEM lowers cost on 72% to 96% of sources.

2. Why cost can change while success barely moves

The success criteria are the standard benchmark thresholds. PushT requires a joint position error of agent and block below 20 pixels and an angle error below 20 degrees, Two-Room a goal distance below 16 pixels, and Reacher both joint errors below 0.05 rad. As the paper argues, early rounds decide whether a plan reaches the goal region, which these thresholds record, and later rounds refine within it, which mainly the cost records. Stopping native CEM at 20 instead of 60 rounds raises cost on 98% of PushT sources and its mean by 31.5%, but lowers success by only 0.56 points. At one quarter of the tolerance it lowers success by 7.76 points. On Two-Room the mean rises by 45.2% and success falls from 2,483 to 2,430 (p=2.4×10⁻⁶). So a large enough cost gap changes success, and the stricter the criterion, the more of the refinement it sees.

3. E2L under stricter success criteria

Yes, and the advantage of E2L-CEM at comparable planning time grows as the criterion tightens. PushT runs all 50 steps after success and saves every state, so we rejudged the 3,375 trajectories of Table 2 (new Appendix S). We fixed the threshold scales 1, 3/4, 1/2 and 1/4 before computing.

Comparison1×3/41/21/4
E14/8 vs R20−0.30+0.21+0.03+2.87 [1.45, 4.27]
E24/8 vs R35−0.24−0.21−0.09−0.59 [−1.87, 0.68]
E36/8 vs R45+0.21+0.24+0.59+1.13 [0.03, 2.16]
R20 vs R60−0.56−1.36−2.96−7.76 [−9.27, −6.16]

Entries are success differences in points, with 95% paired bootstrap intervals. Native CEM with 60 rounds solves 92.3%, 90.2%, 84.0% and 53.9% at the four tolerances. At one quarter of the tolerance, low-budget E2L-CEM solves 2.87 points more than native CEM with 20 rounds in 3.5% less time, and high-budget E2L-CEM solves 1.13 points more than native CEM with 45 rounds. Middle-budget E2L-CEM matches native CEM with 35 rounds at every tolerance in 12.5% less time. High-budget E2L-CEM matches native CEM with 60 rounds at every tolerance, 54.0% against 53.9% at one quarter, in 24% less time.

So stopping earlier gives up the refinement that a strict criterion rewards, and E2L-CEM, which keeps it with local updates, solves more at comparable planning time.

Thank you again for the detailed response and additional experiments. They have addressed my remaining concerns, and I will maintain my current positive rating.

Reviewer_KftiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies where test-time compute in population-based world-model planning actually matters. Focusing on CEM/iCEM/MPPI with frozen latent world models, the authors show that early rounds are critical for decision quality whereas late rounds can be approximated or removed with little or no loss in success across PushT, Reacher, and Two-Room. They propose Exact-to-Local CEM (E2L-CEM), which fits a small linear local response operator around the proposal mean from a few exact “sentinel” queries and reuses it in later rounds, yielding modest wall-clock savings without detectable loss in success for some schedules, and present extensive stage-wise substitution and matched-noise controls to support their claims.

Critical strengths
  1. Well-controlled experiments. The paper replaces evaluations at specific search stages while keeping the world model and planning rules fixed. Paired comparisons, noise controls, and early-stopping experiments provide useful evidence about how different stages affect planning outcomes.
  2. Careful evaluation of practical benefits. The paper reports actual runtime alongside model queries, task success, and final planning cost. Comparisons with simple alternatives, such as fewer search rounds or fewer candidates, clearly expose the trade-off between speed and planning quality.
Critical weaknesses
  1. The conceptual novelty over prior evaluation scheduling work is insufficiently established. CEM (Moss, 2020) already motivates allocating more exact evaluations to early CEM iterations. Although the present paper cites this work, it does not sufficiently clarify what new principle is established beyond this existing work.
  2. The paper does not clearly explain when and why early approximation is harmful. Early approximation actually improves success on Reacher, while stronger random noise also causes more damage when introduced early. Thus, it remains unclear how much of the observed effect comes from the proposed approximation and how much comes from early search being sensitive to errors in general.
  3. The practical advantage over simple early stopping is limited. In the main PushT experiment, E12 reduces runtime by about 3%, while stopping CEM two rounds earlier saves about 12% without a statistically significant drop in task success.
Questions for authors
  1. Can the diagnostic predict whether early approximation will be harmful on new evaluation cases, with predictions fixed before running the approximation experiments?
  2. Under the same runtime budget, when does E2L offer an advantage over simply running fewer CEM rounds? Does its better preservation of model cost translate into better task performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Kfti (1/3)

Thank you for the careful reading, for recognizing the experimental control and runtime reporting, and for the specific questions. Your requests for a prospective prediction and an equal-time comparison led to our two main new experiments.

1. Novelty relative to prior work on scheduling evaluations.

We believe the paper establishes what prior work had not. Moss (2020) proposes allocating more exact evaluations early, but its own scheduling experiment favored uniform allocation, so its premise was never shown. We fix the model, the objective and the update rule in closed-loop world-model planning, and measure how the precision of evaluation at each stage changes the search and the final outcome on three tasks and three planners (Table 1). This adds two findings.

First, whether a ranking is accurate and whether it changes the search are different questions, as FrP4 also notes from Table 6 of the submission. A changed elite set moves the mean in proportion to the proposal scale σ_r, so ranking errors matter less once the proposal has contracted. The abstract now reads:

"The contraction accounts for the asymmetry: a selection error shifts the proposal in proportion to its scale, so on PushT the displacement caused by the operator falls 2.7-fold across rounds while its rank correlation stays at 0.21–0.28."

Section 6.1 gives the per-round numbers, and late substitution does not change success detectably (Table 1).

Second, the operator approximates the predicted representation and scores it with the original objective. At equal queries it keeps about three times the exact-elite overlap of a scalar cost surrogate (Appendix I), and the CEM schedule built on it improves the cost and time trade-off over truncated native CEM at comparable planning time (Section 6.4).

Related Work now reads:

"Moss allocates more exact evaluations to early CEM rounds based on covariance contraction and the influence of early elites. Uniform allocation performed better in its scheduling experiment. We measure the asymmetry directly, by paired stage-wise substitution in closed-loop world-model planning with task success as the outcome, and separate a surrogate's ranking accuracy from its effect on the search: on PushT the operator's rank correlation stays flat across rounds, while the displacement it causes shrinks with the proposal scale. We turn the asymmetry into a gain in the quality–time trade-off and prospectively test whether the native rounds curve predicts sensitivity to early approximation."

So our paper turns the premise behind Moss's allocation into a measured and explained result, and builds a method on it.

2. When and why early approximation is harmful.

Early approximation is harmful because the early proposal is still wide, so the same change in the elite set pushes the mean farther, and every later round inherits this displacement. Section 6.1 now reads:

"The displacement this ranking error causes shrinks with the proposal scale, because a changed elite set shifts the mean in proportion to σ_r (Appendix A): the distance between the updates induced by predicted and exact scores falls by a factor of about 2.7 on PushT and from 1.81 to 0.43 on Reacher (50 fresh sources, σ̄_r from 1.00 to 0.25). Later rounds inherit this displacement, so exact evaluations matter most before the proposal has contracted."

Within a setting, the mean harm comes from a few decisions whose cost rises sharply: on 100 Two-Room CEM sources the mean rises by +16.7 [6.2, 29.4], while the median changes by −0.17 (Table 11, Appendix K).

The operator and general early sensitivity. Section 6.2 now reads:

"On 2,000 confirmation sources, early noise at the top of its calibration grid (standard deviation 256 times the within-round cost spread, Appendix C), whose drift on 20 calibration sources was 4.44 against 5.18 for the operator, also lowers success, from 1842 to 1814 (Holm p=0.0051), and raises first-decision cost by +2.99 [2.13, 3.84]. Early operator substitution solves 1783 (Holm p=2.1×10⁻⁶; p=0.010 against noise) and raises it by +4.85 [3.41, 6.32]."

So perturbing early selection lowers success by itself, and the operator we tested harms the search more than this noise control.

Reacher. The large all-decision cost difference on Reacher comes from replanning after success: the episode ends once both joints enter the threshold and the environment resets to a random state, and 153 of the 155 second decisions with cost above 10 follow such a reset. Under pre-success cost, the estimate on new sources is +4.82, interval −0.02 to 12.13. Section 6.2 now reads: "Under pre-success cost, Reacher shows no detectable early harm, and on 100 new sources the success gain does not recur (178 against 177 of 200; Appendix K). Its success criterion is a distance threshold that need not track model cost."

So Reacher marks a class of tasks in which a change in model cost need not become a change in threshold success.

AuthorsOpenReview ↗

Response to Reviewer Kfti (2/3)

3. Practical advantage over simple early stopping.

The revision compares E2L-CEM directly with truncated native CEM on the same sources at comparable planning times. At the middle budget on PushT, E2L-CEM uses 12.5% less time with no detectable difference in cost. At the low budget on Two-Room, its cost is lower by 6.37 with 4.7% less planning time. At the high budget, its cost is lower than native CEM with 45 rounds, with 1.4% and 5.4% more planning time, and its observed mean cost is close to the full 60-round search with 24% and 7% less time.

The E12 against R13 comparison in the submission matched exact queries. E12 takes 0.152 s per decision and R13 takes 0.138 s. Nearly equal query counts (7,825 against 7,801) do not give equal planning time, which also includes fitting and local updates. The new experiments therefore choose comparison points by complete planning time and test whether the added local refinement improves the cost and time trade-off. The original comparison remains in Appendix E.

Setup of Section 6.4 and Table 2. E2L-CEM uses the periodic-refit schedule E$k$/8: the first k rounds are exact, the local operator is then refitted every 8 rounds from 24 exact sentinel queries, and the last round is exact. E24/8 was selected at R=60 on 200 development sources. The low and high budgets keep its proportion, k=round(0.4R). At each budget, the paired round counts are chosen from planning time alone on 25 disjoint timing sources.

TaskBudgetE2L vs nativeSuccesses E2L / native, McNemar pPre-success cost difference, 95% intervalTime ratio
Two-RoomlowE16/8 R=40 vs R202,415 / 2,430, 0.13−6.37, −7.45 to −5.310.953
Two-RoommiddleE24/8 R=60 vs R362,461 / 2,466, 0.64+0.81, 0.12 to 1.510.878
Two-RoomhighE40/8 R=100 vs R452,483 / 2,477, 0.53−1.49, −2.13 to −0.831.054
PushTlowE14/8 R=36 vs R203,086 / 3,096, 0.25+0.06, −0.23 to 0.370.965
PushTmiddleE24/8 R=60 vs R353,104 / 3,112, 0.28−0.01, −0.25 to 0.220.875
PushThighE36/8 R=90 vs R453,126 / 3,119, 0.31−0.47, −0.64 to −0.301.014

Two-Room uses 2,950 sources, N=150, RTX 6000D. PushT uses 3,375 sources, N=300, RTX 5090. At the middle budget both planners run all confirmation sources on the same GPU. The low- and high-budget E2L arms run on the other GPU, and their time ratios are measured with both planners on one GPU over 25 disjoint timing sources and then applied to the confirmation runs (Table 2 footnote, Appendix C). The smallest Holm-adjusted success p is 0.266 on Two-Room and 0.735 on PushT.

At the middle budget on Two-Room the cost is 0.81 higher, but this point uses 12.2% less planning time, and no significant success difference was detected. Section 6.4 now reads:

"At the high budget, E2L-CEM lowers pre-success cost against native CEM with 45 rounds by 0.47 [0.30, 0.64] on PushT and 1.49 [0.83, 2.13] on Two-Room, using 1.4% and 5.4% more planning time. Its observed success and mean pre-success cost are close to those of native CEM with 60 rounds (PushT 3,126 against 3,115 successes, cost 9.31 against 9.28; Two-Room 2,483 against 2,483, 42.06 against 42.14), in 0.436 against 0.577 s and 0.560 against 0.604 s per decision."

Development runs selected the same schedule, E24/8, independently on both tasks. So at comparable budgets, the added local updates recover most of the model cost lost by truncating at R45, which is the practical advantage of E2L-CEM over simple early stopping.

AuthorsOpenReview ↗

Response to Reviewer Kfti (3/3)

4. Whether the diagnostic predicts early harm on new cases before approximation is run.

We ran the prospective test you asked for (Appendix K.1, Table 16). All eleven predictions were fixed before substitution, and seven agreed with the cost-based outcome.

Rule, identical for every setting. Run native search for R and for ⌈R/2⌉ rounds on the same sources. Predict early sensitivity when the source-clustered 95% interval of the final-cost increase lies above zero. Harm from early substitution is judged by the same criterion on early substitution minus native. Both native runs and the prediction went into a timestamped registration file.

New settingTruncated minus fullPredictionEarly substitution minus nativeHolds
PushT CEM N300, R15+8.93, 5.02 to 13.96sensitive+1.23, −2.78 to 3.95no
PushT CEM N1200, R15+5.89, 2.64 to 10.05sensitive+7.27, 2.10 to 14.31yes
PushT CEM N300, R30+5.14, 2.89 to 8.58sensitive+6.78, −2.74 to 16.89no*
PushT CEM N600, R30+0.43, −3.10 to 3.07not sensitive+5.54, −1.60 to 12.12on cost**
PushT CEM N1200, R30+1.98, 1.23 to 2.87sensitive+13.31, 5.47 to 22.70yes
PushT CEM N2400, R30+1.60, −0.33 to 3.96not sensitive+16.65, 8.10 to 26.88no
PushT CEM N300, R60+1.49, 1.14 to 1.88sensitive+16.36, 14.57 to 18.20yes
Two-Room CEM N150, R30+20.43, 15.20 to 26.37sensitive−7.50, −17.88 to 1.27no
Two-Room CEM N600, R30+7.35, 4.69 to 10.52sensitive+10.77, 1.36 to 22.37yes
Two-Room CEM N1200, R30+5.83, 2.91 to 9.52sensitive+20.47, 9.08 to 34.15yes
Two-Room CEM N300, R60+2.80, 2.18 to 3.54sensitive+47.63, 41.78 to 53.58yes

The nine R=15 and R=30 settings use 100 sources each and the two R=60 settings use 2,550 PushT and 1,125 Two-Room sources. All four disagreements are on 100-source settings. Agreement is judged on cost, over each arm's decisions before its own first success, and on all decisions for PushT N600 R30. Success is reported separately. In the R60 Two-Room test, early substitution reduced success from 1,034 to 916 of 1,125 sources, 139 / 21 discordant pairs, p<0.001. *Successes fell from 94 to 85 of 100, McNemar 13 / 4, p=0.049. **Successes fell from 93 to 84 of 100, 11 / 2, p=0.022. The Two-Room N150 disagreement recurred on 100 independent sources.

Appendix K.1 now reads:

"The two R=60 settings of Table 1, PushT (2,550 sources) and Two-Room (1,125 sources), were predicted sensitive from small truncation effects (+1.49 [1.14, 1.88] and +2.80 [2.18, 3.54]), and early substitution raised the final cost by +16.36 [14.57, 18.20] and +47.63 [41.78, 53.58]."

You are right that, for most earlier settings in the submission, substitution outcomes were known before the prediction. Table 15 shows that on these 10 settings the same rule agrees in 8, with both disagreements on Reacher. Appendix K.1 reports this as agreement with known outcomes, separately from the prospective test in Table 16.

The disagreements also show a pattern. In the PushT R=30 series with N of 300, 1,200 and 2,400, the truncation effect falls from +5.14 to +1.98 and +1.60, while the harm of early substitution grows from +6.78 to +13.31 and +16.65. How much native search still improves and how far early approximation pulls the search are two different quantities.

So the rule uses native search only, predicts before any substitution runs, and held in 7 of 11 new settings, including both of the largest. Our conclusions on early sensitivity rest on direct stage-wise substitution, and the E2L-CEM results in Section 6.4 do not depend on this rule.

5. When E2L beats fewer CEM rounds at equal running time, and whether cost gains become task performance.

At comparable running time, E2L-CEM gains in the trade-off between model cost and time (Table 2): 6.37 lower cost with 4.7% less time at the low Two-Room budget, and 0.47 and 1.49 below R45 at the high budgets.

Longer search does raise success once the budget range is wide enough. Within each paired budget the two arms differ by at most 15 episodes, and no difference is significant. Across budgets, Section 6.4 now reads: "Across the wider budget range, lower cost goes with more successes: from 20 to 60 rounds, native CEM solves 2,430 against 2,483 Two-Room episodes (36 / 89, p=2.4×10⁻⁶) and 3,096 against 3,115 PushT episodes (38 / 57, p=0.064), and high-budget E2L-CEM solves more than native CEM with 20 rounds on both tasks (p=1.4×10⁻³ on PushT and 6.9×10⁻⁶ on Two-Room; exact McNemar tests outside the Holm family)."

So E2L-CEM approaches the model cost and success of longer native search in less planning time.

Thanks you for the detailed response and the new experiments. The new evidence resolves my concerns. The noise control now separates general early sensitivity from the effect of the operator itself, accounting for roughly half of the success loss, and the Reacher anomaly is explained and does not recur on new sources. The comparison with early stopping is also better designed now that it matches complete planning time, and it shows a real gain on PushT at the high budget. I am willing to raise my overall rating to 3.

Reviewer_4X4DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how test-time computation should be allocated across the iterations of population-based planning with frozen world models. It introduces a training-free local response operator, fitted from a small number of exact model evaluations, both as a diagnostic of local predictability and as a substitute for expensive evaluations during selected search stages (i.e., E2L-CEM). This method is supported by a experiment which shows a task-dependent asymmetry on PushT, Two-Room and Reacher. Finally, in the main PushT comparison, E2L-CEM produces identical success outcomes to native CEM across 200 paired source–seed evaluations, with approximately 13% fewer exact queries and 3% lower planning time.

Critical strengths
  1. The problem of reducing test-time compute of world models is important.
  2. The idea is reasonable, and the experiment shows its effectiveness.
Critical weaknesses
  1. The authors need to compare with [1], which not only scalar-objective surrogates but also evaluation schedules that allocate more true evaluations to early iterations, explicitly motivated by covariance contraction and the influence of early elites. I think [1] has a similar intuition to this paper, although it is not the same world-model planning study and its scheduling experiments actually favor uniform allocation in the tested setting.
  2. The practical advantage of E2L over simple truncation is not sufficient. In Table 2, E12 reduces planning time from 0.157 to 0.152 seconds, whereas stopping native CEM after 13 rounds reaches 0.138 seconds with comparable observed success.
Questions for authors
  1. What exactly was predicted prospectively in the eight settings diagnostic test? Table 14 states that already-observed early-substitution outcomes were used to predict the effect of truncating native search on the same sources. This seems to be different from the proposed use of a rounds curve to predict whether early approximation will be harmful.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 4X4D

Thank you for recognizing the importance of the problem and the effectiveness of the method, and for the specific comments, each of which led to a change in the revision.

1. Comparison with [1].

We take [1] to be Moss (2020), Cross-Entropy Method Variants for Optimization. As you note, it motivates more exact evaluations in early iterations from covariance contraction and early elites, and its scheduling experiment favored uniform allocation in the tested settings. Related Work now reads:

"Moss allocates more exact evaluations to early CEM rounds based on covariance contraction and the influence of early elites. Uniform allocation performed better in its scheduling experiment. We measure the asymmetry directly, by paired stage-wise substitution in closed-loop world-model planning with task success as the outcome, and separate a surrogate's ranking accuracy from its effect on the search: on PushT the operator's rank correlation stays flat across rounds, while the displacement it causes shrinks with the proposal scale. We turn the asymmetry into a gain in the quality–time trade-off and prospectively test whether the native rounds curve predicts sensitivity to early approximation."

On PushT the Spearman correlation is only 0.21 to 0.28 at every round, as FrP4 also notes, yet Section 6.1 shows that the update displacement the operator causes falls about 2.7-fold, and the same amount of late substitution produces no detectable change in success. Appendix I compares the local operator with a scalar surrogate control: the operator approximates the predicted representation and scores it with the original objective, and at equal queries it keeps about three times the exact-elite overlap. The stage interventions cover three tasks and three planner families, and the E2L results in Section 6.4 are for CEM. So [1] motivates early allocation, and our paper measures at which stage and why it holds in world-model planning, and builds on it a method that improves the cost and time trade-off at comparable planning time.

2. Practical advantage of E2L over simple truncation.

Section 6.4, Table 2 and Figure 2 compare E2L-CEM with truncation on the same sources at comparable planning times across three budgets. At the middle budget on PushT, E2L-CEM uses 12.5% less time with no detectable difference in cost. At the low budget on Two-Room, its cost is 6.37 lower with 4.7% less planning time. At the high budget, its cost is 0.47 and 1.49 below native CEM with 45 rounds, with 1.4% and 5.4% more planning time, and its observed mean is close to the full 60-round search with 24% and 7% less time. At the middle budget on Two-Room the cost is 0.81 higher, with 12.2% less time. Within each budget no success difference is significant. Across budgets, lower cost goes with more successes: high-budget E2L-CEM solves 3,126 PushT and 2,483 Two-Room episodes against 3,096 and 2,430 for native CEM with 20 rounds (p=1.4×10⁻³ and 6.9×10⁻⁶). The E12 against R13 comparison you cite matched exact queries. E12 takes 0.152 s per decision and R13 takes 0.138 s, because planning time also includes fitting and local updates. The new experiments therefore choose comparison points by complete planning time, and the original comparison remains in Appendix E. So at comparable budgets, the added local updates recover most of the model cost lost by truncation.

3. What was predicted in advance in the diagnostic test over eight settings.

Your reading of the original table is correct: for most earlier settings, early-substitution outcomes were known before the truncation prediction was written, so the original test checked agreement between two observed effects. The revision replaces it with a prospective test using one rule: run native search for R and ⌈R/2⌉ rounds, and predict early sensitivity when the source-clustered 95% interval of the final-cost increase lies above zero. Under the cost criterion 7 of 11 predictions held (Table 16). Appendix K.1 now reads:

"For each new setting, the native runs and the prediction were registered with a timestamp before any substitution arm ran."

"On eleven new settings (seven PushT CEM, four Two-Room CEM, N from 150 to 2,400, R∈{15,30,60}), the prediction held in seven (Table 16)."

In the two largest settings, R=60 on PushT with 2,550 sources and on Two-Room with 1,125, the rule classified the search as sensitive from small truncation effects, and early substitution raised the final cost by +16.36, interval 14.57 to 18.20, and +47.63, interval 41.78 to 53.58. On the 10 earlier settings the rule agrees in 8, and both disagreements are on Reacher, where truncation effects are smallest and success does not follow model cost. Table 15 reports these as agreement with known outcomes. So the rule, applied before any substitution ran, held in 7 of the 11 new settings, including both of the largest.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for their detailed comments. Kfti recognizes the experimental control and runtime reporting, and 4X4D the importance of the problem and the effectiveness of the method. FrP4 states our central observation accurately: the effect of an approximate evaluation depends on the stage of the search at which it enters. A common multi-fidelity strategy spends cheap evaluations on exploration and reserves exact ones for refinement. In world-model planning we find the opposite: the decision is made early, late rounds refine it, and E2L-CEM makes that refinement cheaper. The revision adds three groups of experiments.

1. At comparable planning times, E2L-CEM improves the trade-off between model cost and time. For FrP4, Kfti and 4X4D. New Section 6.4, Table 2 and Figure 2 pair E2L-CEM with truncated native CEM on the same sources at three budgets, 3,375 sources on PushT and 2,950 on Two-Room. Its schedule E$k$/8 scores the first k rounds and the last round exactly and refits the local operator every 8 rounds in between. Both tasks selected E24/8 on development sources, and timing data alone set each pair. At the high budget, E2L-CEM has lower pre-success cost than native CEM with 45 rounds, by 0.47 [0.30, 0.64] on PushT and 1.49 [0.83, 2.13] on Two-Room. Its observed success and mean cost are close to those of native CEM with 60 rounds, 9.31 against 9.28 on PushT and 42.06 against 42.14 on Two-Room, with 24% and 7% less planning time. At the middle budget on PushT, E2L-CEM uses 12.5% less planning time with unchanged pre-success cost, −0.01 [−0.25, 0.22]. At the low budget on Two-Room, it lowers pre-success cost by 6.37 [5.31, 7.45] with 4.7% less time. Within each budget no success difference is significant. Across budgets, lower cost goes with more successes: high-budget E2L-CEM solves 3,126 PushT and 2,483 Two-Room episodes against 3,096 and 2,430 for native CEM with 20 rounds (p=1.4×10⁻³ and 6.9×10⁻⁶). Section 4 now reads:

"A local round costs a fraction of an exact one, and E2L-CEM spends the saving on a larger R."

So at comparable budgets, spending the late-round time on cheap local refinement recovers most of the model cost that truncation loses.

2. Perturbing early selection by itself harms the search. For Kfti. We added score noise only in the early rounds, with its strength set on 20 calibration sources so that its drift is close to the operator's, 4.44 against 5.18. On 2,000 new PushT sources, this noise lowers successes from 1,842 to 1,814, Holm-adjusted p=0.0051. Early substitution of the operator lowers them to 1,783, Holm-adjusted p=2.1×10⁻⁶, and p=0.010 against noise. So early sensitivity does not come from the operator alone, and the operator harms the search more than this noise.

3. In the largest tests, early approximation derails the search and the same amount of late approximation keeps success. For Kfti and 4X4D. Section 6.2 and Table 1 now open with these two tests:

"The largest tests fix the amount of substitution and move only its position. With N=300 and R=60, early (rounds 0–15) and late (rounds 32–47) substitution each replace 16 rounds with one operator fit. On 2,550 PushT sources, early substitution solves 2,026 against 2,363 for native CEM (381 / 44, p<0.001), and late substitution solves 2,354 (19 / 10, p=0.14) with 23.6% less planning time per decision. On 1,125 Two-Room sources, early substitution solves 916 against 1,034 (139 / 21, p<0.001), and late substitution solves 1,032 (7 / 5, p=0.77) with 25.1% less planning time."

In the prospective test of the rounds-curve rule, both settings were predicted sensitive before substitution ran, from truncation effects of only +1.49 and +2.80. Table 16 in Appendix K.1 gives all 11 new settings. So the same amount of approximation derails the search when placed early, and placed late it keeps success and saves about a quarter of the planning time.

4. Relation to Moss (2020). For Kfti and 4X4D. Introduction, paragraph 2, now reads:

"For CEM, Moss (2020) argues the opposite: early elites shape every later proposal, so exact evaluations belong early. In that study's scheduling experiment, however, uniform allocation performed better. Where a cheap approximation can go, and whether the time it saves buys a better plan, thus has to be measured where each evaluation is a world-model rollout and the outcome is task success."

5. Presentation. For Kfti and 4X4D. Every subsection of Section 6 now opens with its conclusion, and each symbol is defined where it is first used. Section 5 defines the new metric and the direction of McNemar counts:

"Pre-success cost is the mean exact final cost of the decisions an arm takes on a source before it solves it, differenced between arms per source." "We write paired counts as a / b: episodes solved by the native planner alone / by the compared arm alone."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.