- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submission targets exploration in continuous control with many actuators. FOCUS adds a high-level policy that samples Bernoulli "preference prompts" over action dimensions, and a low-level policy conditioned on these prompts outputs the full action vector. Two KL-based regularizers push actions on activated dimensions to differ across prompts while keeping inactive dimensions consistent. Built on DreamerV3, the method also uses a sequential Monte Carlo planner that rolls out several prompts and reweights trajectories by accumulated advantages. Experiments cover four DMC tasks and two MyoSuite tasks from pixels, four HumanoidBench tasks (where the hierarchy and regularizers are transplanted into FastTD3 without a world model or planner), and an IEEE 39-bus grid control task built on SAC. The authors attribute the reported gains to focused exploration over task-relevant control units.
Critical strengths
- The problem is well motivated, and the evaluation reaches beyond standard locomotion to a power-grid control task. The preference visualizations (Figs. 9, 14–16) offer some qualitative indication that the high-level policy responds to task phases. These remain illustrative rather than evidential, however, and do not offset the methodological concerns below.
Critical weaknesses
(W1) What is implemented differs from what is claimed. The paper states that executed actions are not hard-masked, so unselected dimensions still produce outputs and exploration noise, while only enforces consistency across prompts. Search-space reduction is therefore asserted rather than realized. Fig. 15 also suggests that fewer than one dimension is active per step on average, which raises the possibility that the high-level policy has collapsed.
(W2) The gains cannot be attributed to switching. During imagination, the method keeps the highest-value next state among candidates (Alg. 1, line 13), an optimistic best-of- search that DreamerV3 does not perform. The main comparisons also pit a planning agent against non-planning baselines. Ablations cover a single task with largely overlapping error bars.
(W3) The baselines and task selection are not credible. TD-MPC2's scores in Table 2 (, , ) fall far below published results. Dog Run and Humanoid Run are omitted, every experiment uses three seeds, and HumanoidBench evaluates a FastTD3 variant without the world model or planner.
Questions for authors
Q1. Is the 500K HumanoidBench budget counted across all parallel environments or per environment? How many environments do FastTD3 and FastTD3-FOCUS use, and does the DreamerV3-based FOCUS also rely on parallel simulation?
Q2. Could you ablate the max-value candidate selection in imagination, for instance FOCUS with random candidate selection, and DreamerV3 equipped with the same best-of- step?
Q3. How was TD-MPC2 configured? Its results in Table 2 and Fig. 3 are far below the original paper. Please report results with the official hyperparameters or explain the gap, and clarify whether its planner is CEM or MPPI, since the paper uses both terms.
Q4. What is the average number of activated dimensions per step during training? Does performance change if inactive dimensions are hard-masked or their exploration noise is suppressed?
Q5. Could you add Dog Run and Humanoid Run, and repeat the ablations on at least one DMC and one HumanoidBench task with more seeds?
Q6. A formal characterization of when prompt-conditioned exploration reduces sample complexity would substantially strengthen the contribution.
2 comments
Q6: When does prompt-conditioned exploration reduce sample complexity?
Our contribution is primarily empirical and algorithmic rather than theoretical. The intuition is the following. In many high-dimensional tasks, the control units that matter at a given moment form a small, phase-dependent subset. A flat policy must discover useful behavior by varying all dimensions at once. FOCUS concentrates variation on a learned subset and searches over a compact discrete preference space. The benefit should therefore be largest when rewards are sparse and the relevant subset is small relative to the full action space. Our results match this picture: the gains are largest in sparse-reward Humanoid tasks and on KeyTurnHard, and smaller in Dog tasks, where dense rewards make exploration less critical (Section 5.2). A formal characterization is an interesting direction for future work.
Thank you for the constructive review, and we address each point below.
W1 / Q4: Full-dimensional actions and preference collapse
FOCUS intentionally does not hard-mask executed actions. High-dimensional bodies require all actuators to coordinate, and zeroing joints would break basic behaviors such as balance. FOCUS instead structures exploration through its two regularizers. Diversity makes a dimension behave differently when its bit is switched on. Consistency keeps a dimension stable across all prompts in which it is inactive, which suppresses prompt-induced variation on unselected dimensions. When the preference varies, during training or across planning candidates, behavioral variation is therefore concentrated on the selected dimensions. This is the sense of "active subspace" in our paper.
No collapse. The high-level policy is co-optimized with the regularizers (the gradient is not stopped at un), and the diversity term explicitly rewards distinct activations, which counteracts collapse. During training, the average number of activated dimensions per step is about 10 on MyoSuite KeyTurnHard and about 2 on DMC HumanoidWalk. The count therefore adapts to the task rather than collapsing to zero.
W2 / Q2: Training-time candidate selection
Thank you for catching this. The max-value selection was not used in the implementation behind the reported results. Two descriptions in the manuscript are inaccurate:
The actual procedure is as follows. At each imagination step, a single preference is sampled, and its action advances the imagined trajectory. The additional Nc preference candidates are used only to compute Ldiv and Lcst, and only at the first imagination step, as stated in Section 4.3. No best-of-N search is performed in imagination, so the reported gains do not involve this operation.
W3 / Q3: TD-MPC2 configuration and planner
We use the official TD-MPC2 implementation with its default hyperparameters. To keep comparisons fair, all model-based methods share the same observations and training budget. The main DMC experiments (Fig. 3) and Table 2 both use pixel inputs. In Table 2, TD-MPC2, DreamerV3, and FOCUS are all trained for the same 500K environment steps. The published TD-MPC2 numbers mostly use proprioceptive states and substantially longer training, so they are not directly comparable. Under the identical protocol, DreamerV3 and FOCUS reach near-saturated returns, which indicates that the budget is adequate for a strong model-based agent.
Planner. TD-MPC2 uses MPPI, as described correctly in Section 4.4 and Appendix D.2. The phrase "CEM-based planning" in the baseline list (Section 5.1) is a typo. The separate CEM experiment in Fig. 8a is a variant of FOCUS's planner, used to compare SMC with CEM, and is unrelated to TD-MPC2.
Q1: HumanoidBench budget and parallel environments
FastTD3 and FastTD3-FOCUS use the identical setup: 64 parallel environments, each run for 500K steps, so the total budget is 64 × 500K transitions for both methods. The "500K" in Section 5.1 refers to steps per environment, and the comparison is matched. The DreamerV3-based FOCUS is not evaluated on HumanoidBench. In DMC and MyoSuite it uses only one environment, and all baselines in those domains share the same frame budget.
W3 / Q5: Task selection, seeds, and the role of HumanoidBench
The HumanoidBench experiments test whether the hierarchical switching mechanism is a portable component rather than something tied to our world model and planner. We integrate only the high-level preference policy and the diversity/consistency objectives into FastTD3, without a world model or planning, and FastTD3-FOCUS improves sample efficiency over a strong, well-tuned FastTD3 on all four 56-D H1hand tasks. Together with the SAC-based results on power-grid control, this shows that control-unit switching helps high-dimensional control across model-based and model-free learners, as well as across pixel-based and proprioceptive inputs.
Our DMC tasks cover both embodiments (Humanoid, 21-D; Dog, 38-D) and two regimes: sparse-reward tasks where exploration is the bottleneck, and denser-reward tasks where it is not. The Run variants share the same embodiments and action spaces as the tasks we report, and it takes large time budget to conduct the experiments.