- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper revisits the value of test-time planning for continuous control. Its main agent combines a Gaussian-latent world model, imagined actor–critic learning, and annealed curiosity, obtaining strong final results on five DMC and six HumanoidBench tasks at four million environment steps. A planner and imitation temperature sweep shows that planning can help or hurt depending on the task. The results are interesting, but I have concerns about differences between the paper and code, the explanation of the planning diagnostics, and how broadly the comparison can be interpreted.
Critical strengths
- The final performance is strong across the evaluated tasks, particularly on humanoid-run and H1hand-hurdle. Five-seed results and learning curves help assess the gains and reveal the slower start of the proposed method (Table 2; Figure 5).
- The controlled framework includes planner-only settings as well as planner imitation. I appreciate that the full sweep shows large positive results alongside failures, which makes the task dependence clear (Figure 3).
- Using model prediction losses to supply imagined curiosity is a practical design. The large categorical/Gaussian difference is also a useful observation, even though its cause needs more investigation (Section 3.1; Figure 2).
Critical weaknesses
- I could not fully reconcile the method description with the anonymous code. Eq. 6 anneals curiosity using the median estimated extrinsic return, but the available implementation uses the midpoint of percentile bounds from a normalizer updated with curiosity-shaped returns. The imitation implementation also uses running normalization, detaches the policy standard deviation, and has a separate loss coefficient whose setting for the sweep is unclear. Please provide the resolved configurations for the reported runs and update the equations to match them. Unfortunately, these differences make it difficult to tell which precise recipe supports the main conclusions. They do not, on their own, show that the reported results are invalid. See the agent implementation, normalizer, and configuration.
- The interpretation of ESS needs to account for the normalization in Eq. 8. Scaling all candidate return differences by a positive constant leaves the weights and ESS unchanged unless the variance floor becomes active. A flatter absolute value scale therefore does not by itself explain higher ESS. Please examine return spread, standardized distribution shape, and floor activation. Higher trajectory ESS also need not mean more diverse actions or replay states, and gradient norms alone do not show conflicting updates. A delayed-planning experiment with model-error and diversity measurements would help test the proposed explanation. Until then, the causal language should stay explicitly tentative (Section 4.3; Figure 4).
- The planning conclusion is broader than the controlled comparison. The tested planner uses one iteration, horizon eight, policy-prior proposals, and stochastic trajectory selection. The temperature sweep is informative within that design, but it does not establish how another reasonable planning budget would behave. More directly, Figure 2 does not show planning added to the full curiosity agent. A small factorial comparison on representative high-dimensional tasks would clarify whether curiosity replaces planning or works well with it. Please state the reward used for planning and consider whether favorable sampled model outcomes affect candidate ranking (Sections 3.2 and 4.2; Table 3).
- The categorical-latent result needs a more cautious explanation. Figure 2 shows a severe failure of the tested categorical configuration, but a similar parameter count does not match latent capacity, regularization, or optimization. Please provide the full categorical settings and some evidence on prediction quality and latent usage. A modest capacity or KL/free-nat sensitivity check would help establish whether an information bottleneck is responsible. Without that evidence, the claim should be limited to the configurations tested (p. 2; Figure 2).
- The compute comparison remains incomplete. The four-million-step results support strong final performance under a fixed interaction budget, but the runtime is reported only for the proposed method. The optimizer-step entry in Table 3 also needs clarification against the ratio-based training implementation. Please report actual updates, imagined transitions, total training time, and action-selection latency, and identify the source and evaluation settings of the baseline results. This would make the claim about computational scalability easier to assess. See the training loop and configuration.
Questions for authors
- Which exact configurations produced the reported results, and how should the annealing and imitation equations be revised to match them?
- What explains the ESS changes after accounting for return standardization and the variance floor? Does delaying planning improve the degraded runs in the way the proposed mechanism predicts?
- What happens when planning is added to the full curiosity agent, including a modest search-budget or rollout-averaging check?
- Do the categorical results persist under a reasonable capacity and regularization comparison, and what measurements support the bottleneck interpretation?
- What are the actual training and inference costs for each method, and how were the baseline scores obtained and evaluated? Please clarify the optimizer-step count in Table 3.