acceptodds
Withdrawn from ICLR 2027

FOCUS: Scaling Continuous Control via Structured Exploration over Diversified Action Subspaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning in high-dimensional continuous action spaces faces a significant bottleneck: the exponentially expanding search space leads to poor sample efficiency and limited exploration scalability. To address this challenge, we propose FOCUS, a framework designed for structured exploration that avoids the exhaustive search of the full action space. The core insight of FOCUS lies in decoupling control into a hierarchical selection of diversified action subspaces, effectively rendering the exploration process more tractable. Specifically, a high-level policy generates binary preferences to dynamically activate task-critical control units, while a low-level policy produces precise actions conditioned on these selections. To ensure the effectiveness of this hierarchy, we incorporate a diversity-consistency tradeoff in the optimization objective. This objective encourages functional specialization by promoting distinct action distributions between preferences that select a control unit and those that do not, while encouraging consistency across preferences that leave the unit unselected. Extensive evaluations on challenging continuous control benchmarks demonstrate that FOCUS significantly outperforms state-of-the-art methods, particularly as the action dimensionality increases, establishing its superior scalability and exploration efficiency.

Primary area:
reinforcement learning
Keywords:
High-dimensional continuous control, Model-based reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mq9mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The submission targets exploration in continuous control with many actuators. FOCUS adds a high-level policy that samples Bernoulli "preference prompts" over action dimensions, and a low-level policy conditioned on these prompts outputs the full action vector. Two KL-based regularizers push actions on activated dimensions to differ across prompts while keeping inactive dimensions consistent. Built on DreamerV3, the method also uses a sequential Monte Carlo planner that rolls out several prompts and reweights trajectories by accumulated advantages. Experiments cover four DMC tasks and two MyoSuite tasks from pixels, four HumanoidBench tasks (where the hierarchy and regularizers are transplanted into FastTD3 without a world model or planner), and an IEEE 39-bus grid control task built on SAC. The authors attribute the reported gains to focused exploration over task-relevant control units.

Critical strengths
  1. The problem is well motivated, and the evaluation reaches beyond standard locomotion to a power-grid control task. The preference visualizations (Figs. 9, 14–16) offer some qualitative indication that the high-level policy responds to task phases. These remain illustrative rather than evidential, however, and do not offset the methodological concerns below.
Critical weaknesses

(W1) What is implemented differs from what is claimed. The paper states that executed actions are not hard-masked, so unselected dimensions still produce outputs and exploration noise, while Lcst\mathcal{L}^{cst} only enforces consistency across prompts. Search-space reduction is therefore asserted rather than realized. Fig. 15 also suggests that fewer than one dimension is active per step on average, which raises the possibility that the high-level policy has collapsed.

(W2) The gains cannot be attributed to switching. During imagination, the method keeps the highest-value next state among NcN_c candidates (Alg. 1, line 13), an optimistic best-of-NN search that DreamerV3 does not perform. The main comparisons also pit a planning agent against non-planning baselines. Ablations cover a single task with largely overlapping error bars.

(W3) The baselines and task selection are not credible. TD-MPC2's scores in Table 2 (332332, 203203, 162162) fall far below published results. Dog Run and Humanoid Run are omitted, every experiment uses three seeds, and HumanoidBench evaluates a FastTD3 variant without the world model or planner.

Questions for authors

Q1. Is the 500K HumanoidBench budget counted across all parallel environments or per environment? How many environments do FastTD3 and FastTD3-FOCUS use, and does the DreamerV3-based FOCUS also rely on parallel simulation?

Q2. Could you ablate the max-value candidate selection in imagination, for instance FOCUS with random candidate selection, and DreamerV3 equipped with the same best-of-NcN_c step?

Q3. How was TD-MPC2 configured? Its results in Table 2 and Fig. 3 are far below the original paper. Please report results with the official hyperparameters or explain the gap, and clarify whether its planner is CEM or MPPI, since the paper uses both terms.

Q4. What is the average number of activated dimensions per step during training? Does performance change if inactive dimensions are hard-masked or their exploration noise is suppressed?

Q5. Could you add Dog Run and Humanoid Run, and repeat the ablations on at least one DMC and one HumanoidBench task with more seeds?

Q6. A formal characterization of when prompt-conditioned exploration reduces sample complexity would substantially strengthen the contribution.

2 comments

Thank you for the constructive review, and we address each point below.

W1 / Q4: Full-dimensional actions and preference collapse

FOCUS intentionally does not hard-mask executed actions. High-dimensional bodies require all actuators to coordinate, and zeroing joints would break basic behaviors such as balance. FOCUS instead structures exploration through its two regularizers. Diversity makes a dimension behave differently when its bit is switched on. Consistency keeps a dimension stable across all prompts in which it is inactive, which suppresses prompt-induced variation on unselected dimensions. When the preference varies, during training or across planning candidates, behavioral variation is therefore concentrated on the selected dimensions. This is the sense of "active subspace" in our paper.

No collapse. The high-level policy is co-optimized with the regularizers (the gradient is not stopped at unu_n), and the diversity term explicitly rewards distinct activations, which counteracts collapse. During training, the average number of activated dimensions per step is about 10 on MyoSuite KeyTurnHard and about 2 on DMC HumanoidWalk. The count therefore adapts to the task rather than collapsing to zero.

W2 / Q2: Training-time candidate selection

Thank you for catching this. The max-value selection was not used in the implementation behind the reported results. Two descriptions in the manuscript are inaccurate:

  1. Section 4.3 ("we take st+1,ns_{t+1,n} … with the highest predicted value … as st+1s_{t+1}") is incorrect.
  2. Algorithm 1, the step "Select st+1,ns_{t+1,n} with highest v(st+1,n)v(s_{t+1,n}) as st+1s_{t+1}", should be removed.

The actual procedure is as follows. At each imagination step, a single preference is sampled, and its action advances the imagined trajectory. The additional NcN_c preference candidates are used only to compute Ldiv\mathcal{L}_{\text{div}} and Lcst\mathcal{L}_{\text{cst}}, and only at the first imagination step, as stated in Section 4.3. No best-of-NN search is performed in imagination, so the reported gains do not involve this operation.

W3 / Q3: TD-MPC2 configuration and planner

We use the official TD-MPC2 implementation with its default hyperparameters. To keep comparisons fair, all model-based methods share the same observations and training budget. The main DMC experiments (Fig. 3) and Table 2 both use pixel inputs. In Table 2, TD-MPC2, DreamerV3, and FOCUS are all trained for the same 500K environment steps. The published TD-MPC2 numbers mostly use proprioceptive states and substantially longer training, so they are not directly comparable. Under the identical protocol, DreamerV3 and FOCUS reach near-saturated returns, which indicates that the budget is adequate for a strong model-based agent.

Planner. TD-MPC2 uses MPPI, as described correctly in Section 4.4 and Appendix D.2. The phrase "CEM-based planning" in the baseline list (Section 5.1) is a typo. The separate CEM experiment in Fig. 8a is a variant of FOCUS's planner, used to compare SMC with CEM, and is unrelated to TD-MPC2.

Q1: HumanoidBench budget and parallel environments

FastTD3 and FastTD3-FOCUS use the identical setup: 64 parallel environments, each run for 500K steps, so the total budget is 64 × 500K transitions for both methods. The "500K" in Section 5.1 refers to steps per environment, and the comparison is matched. The DreamerV3-based FOCUS is not evaluated on HumanoidBench. In DMC and MyoSuite it uses only one environment, and all baselines in those domains share the same frame budget.

W3 / Q5: Task selection, seeds, and the role of HumanoidBench

The HumanoidBench experiments test whether the hierarchical switching mechanism is a portable component rather than something tied to our world model and planner. We integrate only the high-level preference policy and the diversity/consistency objectives into FastTD3, without a world model or planning, and FastTD3-FOCUS improves sample efficiency over a strong, well-tuned FastTD3 on all four 56-D H1hand tasks. Together with the SAC-based results on power-grid control, this shows that control-unit switching helps high-dimensional control across model-based and model-free learners, as well as across pixel-based and proprioceptive inputs.

Our DMC tasks cover both embodiments (Humanoid, 21-D; Dog, 38-D) and two regimes: sparse-reward tasks where exploration is the bottleneck, and denser-reward tasks where it is not. The Run variants share the same embodiments and action spaces as the tasks we report, and it takes large time budget to conduct the experiments.

Q6: When does prompt-conditioned exploration reduce sample complexity?

Our contribution is primarily empirical and algorithmic rather than theoretical. The intuition is the following. In many high-dimensional tasks, the control units that matter at a given moment form a small, phase-dependent subset. A flat policy must discover useful behavior by varying all dimensions at once. FOCUS concentrates variation on a learned subset and searches over a compact discrete preference space. The benefit should therefore be largest when rewards are sparse and the relevant subset is small relative to the full action space. Our results match this picture: the gains are largest in sparse-reward Humanoid tasks and on KeyTurnHard, and smaller in Dog tasks, where dense rewards make exploration less critical (Section 5.2). A formal characterization is an interesting direction for future work.

Reviewer_kRhGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FOCUS, a framework for high-dimensional continuous control that conditions a full-dimensional actor on state-dependent binary preferences over control units. Experiments on multiple environments show improved learning efficiency and performance.

Critical strengths

The method addresses a clear practical problem: exploring all action dimensions at once can make learning inefficient. FOCUS lets the agent learn which control units to focus on in each state, while still allowing all units to coordinate their actions. The regularization gives the low-level policy a reason to follow these preferences instead of ignoring them.

Critical weaknesses
  1. It is unclear whether the gains come from focusing exploration. Unselected action dimensions still produce actions and can remain highly random. The preference visualizations show which units are selected, but do not establish that these units matter more for the task. Without comparisons to random preferences and a similarly sized flat policy, the gains could come from the larger policy or the added regularization.

  2. The experiments do not convincingly establish the claimed scalability. Comparing different tasks with different action dimensions cannot isolate the effect of dimensionality, since the tasks also differ in difficulty and reward structure. A controlled experiment that increases action dimensions within the same task is missing. Comparisons with closely related exploration methods, particularly DEP-RL and Lattice, are also needed to assess the advantage over existing approaches.

Questions for authors
  1. Implementation clarification: How are gradients propagated through the sampled Bernoulli preferences and the indicator functions in the regularizers? For FastTD3-FOCUS, how are the action distributions used in the KL objectives defined for the deterministic actor?

  2. Can the authors provide a controlled action-dimension experiment and comparisons with DEP-RL or Lattice under comparable interaction and computational budgets?

2 comments

Thank you for the constructive review, and we address each point below.

W1: Are the gains due to focused exploration, rather than extra capacity or regularization?

We isolate these factors on MyoSuite KeyTurnHard:

MethodEpisode return (mean ± std)
FOCUS (full)6030 ± 1856
FOCUS w/o planning4690 ± 1889
Capacity-matched DreamerV33519 ± 2704
DreamerV33281 ± 2346

Capacity. A capacity-matched flat DreamerV3, whose actor is enlarged to match FOCUS without the hierarchical preference mechanism, improves only slightly over DreamerV3 (3519 vs. 3281). It remains well below FOCUS even without planning (4690). Extra actor capacity therefore does not explain the gain.

Planning. FOCUS without planning still clearly outperforms both flat baselines. Conversely, adding execution-time planning to a flat DreamerV3 (DreamerV3-Planning, Fig. 5b) is unstable and prone to local optima. Planning alone therefore does not explain the gain either. Planning helps most when it searches over the preference space that FOCUS learns.

Regularization vs. learned preferences. Fig. 5b ablates both components separately. Removing the switching policy (keeping the rest of the pipeline) degrades performance below DreamerV3. Removing the diversity/consistency objectives also causes a clear drop. Neither component is sufficient alone. The gain comes from their combination: a learned high-level policy whose bits are given functional meaning by the regularizers. The learned preferences are also clearly state-dependent: Figs. 9 and 14 show selections switching with task phase, which a fixed or random prompt cannot produce.

W2 / Q2: Scalability and comparisons with DEP-RL / Lattice

By scalability, we mean that FOCUS outperforms strong flat baselines on high-dimensional control problems (21-D to 56-D action spaces across four benchmark families and three host algorithms), which is the regime where uniform exploration is the bottleneck. As noted in our reply to R1, varying action dimension within one task requires artificially duplicating or padding actuators. This yields redundant dimensions unlike the coupled, phase-dependent structure of real embodiments, so we evaluate on natively high-dimensional systems.

DEP-RL and Lattice both target musculoskeletal control by redesigning the exploration noise. DEP-RL alternates the policy with a DEP controller, which requires one proprioceptive sensor per actuator (muscle length/force) and a known sensor–actuator correspondence. Lattice injects correlated noise through the policy's latent layer in model-free agents operating on proprioceptive state. FOCUS operates at a different level: it changes the policy structure (a high-level selection of control units) and the planning space (search over preferences), and it learns from pixels in our main experiments. These approaches are complementary. A correlated noise process could in principle be used inside the FOCUS low-level policy.

Q1: Gradients through the Bernoulli samples and the indicator functions

We use a straight-through estimator. Preferences are sampled as hard binary vectors in the forward pass, and gradients flow back through the Bernoulli probabilities to the high-level policy. The indicator weights in Eqs. (3)–(4) are computed from these sampled preferences and share the same straight-through path. This is why we do not stop the gradient at unu_n (Section 4.3): the regularizers also train the high-level policy, and this prevents it from collapsing to identical activation patterns.

Q1: KL objectives for the deterministic FastTD3 actor

The TD3 actor update is unchanged and remains a deterministic policy gradient. Given a preference uu, the low-level network defines a Gaussian π(a∣s,u)=N(aˉθ(s,u), σθ(s,u))\pi(a \mid s,u)=\mathcal{N}(\bar a_\theta(s,u),\, \sigma_\theta(s,u)) with mean aˉθ(s,u)=tanh⁡μθ(s,u)\bar a_\theta(s,u) = \tanh \mu_\theta(s,u). Only the mean aˉθ(s,u)\bar a_\theta(s,u) enters the actor loss and the environment. The critic target is still this mean plus TD3's clipped target noise. We do not sample from the Gaussian or add an entropy term, so the distributional Bellman backup is exactly that of FastTD3.

The two KL terms are auxiliary regularizers on this Gaussian and do not change the behavior policy. For a minibatch of states, we draw NcN_c preferences from the high-level Bernoulli head. Along each action dimension we form a positive pair (same bit, u+u^{+}) and a negative pair (opposite bit, u−u^{-}), and optimize

Lactor=− Q(s,aˉθ(s,u))+λmin⁡ KL(π(⋅∣s,u) ∥ π(⋅∣s,u+))−λmax⁡ KL(π(⋅∣s,u) ∥ π(⋅∣s,u−)).\mathcal{L}_{\text{actor}} = -\,Q\big(s, \bar a_\theta(s,u)\big) + \lambda_{\min}\,\mathrm{KL}\big(\pi(\cdot\mid s,u)\,\|\,\pi(\cdot\mid s,u^{+})\big) - \lambda_{\max}\,\mathrm{KL}\big(\pi(\cdot\mid s,u)\,\|\,\pi(\cdot\mid s,u^{-})\big).

The counterpart distribution is stop-gradient. Because the KL depends on both mean and variance, the first term pulls the means of matching preferences together and the second pushes the means of opposing preferences apart. This is well defined for a deterministic controller: aˉθ(s,u)\bar a_\theta(s,u) is the location of π\pi, and the KL only shapes how these locations are arranged across preferences. σθ\sigma_\theta is constrained to [0.1,1.0][0.1, 1.0] and is never executed. It shares the trunk with the mean head, so its gradient can shape shared features, but the action that TD3 differentiates through QQ is always the mean.

In short, the stochasticity in FastTD3-FOCUS lies in the high-level preference uu (Bernoulli, straight-through gradient). Once uu is given, the motor command is the deterministic mean.

Reviewer_aLTiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FOCUS, a model-based RL framework for continuous control with high-dimensional action spaces. A high-level policy samples binary preferences over action dimensions, and a low-level policy produces the full action conditioned on these preferences. Two additional losses are introduced : a diversity loss that maximizes the KL divergence of an action dimension’s distribution between prompts where it is active and where it is not, and a consistency loss that keeps it unchanged across prompts where it is inactive. At interaction time, the agent plans by sampling several preference prompts, rolling them out in the learned world model, and weighting them by advantage. FOCUS is evaluated on pixel-based DMC (Humanoid, Dog) and MyoSuite. FastTD3- and SAC-based variants are evaluated on HumanoidBench and a power-grid control task with improvements.

Critical strengths
  1. The idea of learning, per state, which control units matter is interesting and well motivated, and the selection visualizations (Figs. 14–16) look reasonable.
  2. The mechanism is fairly general: the authors plug it into model-based and model-free algorithms, and it shows gains in all three settings.
  3. The proposed planner is cheaper to run compared to baseline model-based methods.
Critical weaknesses
  1. The "structured exploration" and "active subspace" claims don’t match the method. There is no clear definition in the paper, but from the Eq.2, the binary prompt is only an extra input to the low-level actor; the executed action is never masked (line 109), and all dimensions keep their entropy noise. The link between bit i and dimension i comes only from the two losses. Fig. 1(3) depicts a downsampling to a subspace that the method does not perform.
  2. Better exploration is claimed but not shown. There’s no direct measure, such as per-dimension action variance or state coverage, and no controlled study over action dimensionality to back up the scaling claim. Gains are actually larger on Humanoid (21-D) than on Dog (38-D).
  3. The baselines look weak. TD-MPC2 scores poorly in the Table 2, which suggests a tuning or implementation issue. Action-reduction methods the paper cites (e.g., Stolz et al. 2024) and DEP-RL for MyoSuite are not compared.
  4. From the writing, several method details are unclear or unjustified. It isn’t said how gradients pass through the binary samples into the high-level policy.
Questions for authors
  1. How exactly is u (preference) fed to the interaction policy, and does an inactive bit ever constrain the executed action? If not, in what sense is exploration restricted to a subspace?
  2. How are gradients propagated through the Bernoulli samples to π_φ1 (straight-through, Gumbel, REINFORCE)?
  3. Can you explain why TD-MPC2 is performing so poorly in the Table. 2 compared to the paper's original reported numbers?
1 comment

Thank you for the constructive review, and we address each point below.

W1 / Q1: How is the preference fed to the policy, and in what sense is exploration structured?

The binary preference uu is concatenated with the latent state and fed to the interaction policy πθ2(a∣s,u)\pi_{\theta_2}(a \mid s, u), which outputs a full-dimensional action distribution. As stated in Section 2.1 (line 109), an inactive bit does not mask the executed action. High-dimensional bodies need all actuators to coordinate, so hard-zeroing joints would break basic behaviors such as balance.

FOCUS structures where behavioral variation comes from; it does not shrink the executable action space. The two regularizers give each bit a functional meaning:

  • Diversity (Ldiv\mathcal{L}_{\text{div}}) makes dimension ii behave differently when uiu_i is switched on, so toggling a bit produces distinct behavior along that dimension.
  • Consistency (Lcst\mathcal{L}_{\text{cst}}) keeps dimension ii consistent across all prompts in which it is inactive. This suppresses prompt-induced diversity on dimensions that are not selected.

Consequently, when the agent varies its preference (stochastically during training, or across candidates in preference-based planning), the resulting behavioral changes are concentrated on the selected dimensions, and the remaining dimensions keep a stable, coordinated behavior. This is the sense of "active subspace" in our paper: the subspace in which exploration varies, not a projection of the executed action. Fig. 1(3) is a schematic of this concentration of variation. It is not a literal downsampling of the action vector.

W2: Evidence for better exploration and scalability

By scalability, we mean that FOCUS handles high-dimensional control problems substantially better than strong flat baselines, which is the practical bottleneck we target. The evidence spans 21-D to 56-D action spaces, four benchmark families (pixel-based DMC and MyoSuite, proprioceptive HumanoidBench, and power-grid control), and three host algorithms (DreamerV3, FastTD3, SAC).

On exploration specifically, the clearest signal is in the sparse-reward Humanoid tasks (Section 5.2). All methods go through a long warm-up before reaching rewarding regions, and FOCUS escapes it markedly earlier than DreamerV3 (Fig. 3). The preference visualizations (Figs. 9, 14) show that selections change with task phase rather than staying fixed.

The Humanoid-vs-Dog comparison is consistent with this explanation rather than contradicting it. As analyzed in Section 5.2, Dog rewards are dense enough that DreamerV3 already reaches high-reward regions steadily, so exploration is not the bottleneck there. FOCUS gains most where exploration in a large action space is the bottleneck, and that is exactly what the method is designed for.

W3 / Q3: TD-MPC2 performance and additional baselines

TD-MPC2 configuration. We use the official TD-MPC2 implementation with its default hyperparameters. To keep the comparison fair, all model-based methods receive the same observations and the same training budget. Table 2 reports pixel-based DMC results, and TD-MPC2, DreamerV3, and FOCUS are all trained for the same 500K environment steps. The numbers in the original TD-MPC2 paper mostly come either from proprioceptive state inputs or much longer training. Learning from pixels under a 500K budget is a substantially harder regime, and TD-MPC2 has not converged at that point. DreamerV3 and FOCUS reach near-saturated returns under the identical protocol, which shows the budget is adequate for a strong model-based agent and the setting is not biased against model-based methods.

Action-reduction methods and DEP-RL. Action-masking approaches such as Stolz et al. (2024) restrict the action space to a state-dependent relevant action set that must be supplied or derived from task-specific knowledge. FOCUS addresses the setting in which no such knowledge is available, and learns which control units to prioritize directly from reward. DEP-RL targets overactuated musculoskeletal systems. Its DEP exploration controller requires one proprioceptive sensor per actuator (e.g., muscle length/force) with a known sensor–actuator correspondence. Our MyoSuite and DMC experiments learn from pixels, where this actuator-paired input is not part of the observation. DEP-RL also changes the exploration noise process, whereas FOCUS changes the policy structure and planning space. The two are complementary rather than competing.

W4 / Q2: Gradients through the Bernoulli preferences

We use a straight-through estimator. The forward pass uses hard binary samples, and the backward pass propagates the gradient through the Bernoulli probabilities to θ1\theta_1. This is the gradient path meant by "we do not stop the gradient at unu_n" in Section 4.3. It is what allows the high-level policy to be co-optimized with the regularizers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.