acceptodds
Under review as a conference paper at ICLR 2027

K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agents

Abstract

Parallelization in Reinforcement Learning is typically used to speed up the training of a single policy, where multiple workers collect experience from the same sampling distribution. This common design limits the potential of parallelization by neglecting the advantages of diverse exploration strategies. We propose K-Myriad, a scalable and unsupervised method that maximizes the collective state entropy induced by a population of parallel policies. By cultivating a portfolio of specialized exploration strategies, K-Myriad provides a robust initialization for Reinforcement Learning, leading to both higher training efficiency and the discovery of heterogeneous solutions. Experiments on high-dimensional continuous control tasks, with large-scale parallelization, demonstrate that K-Myriad can learn a broad set of distinct policies, highlighting its effectiveness for collective exploration and paving the way towards novel parallelization strategies.

Primary area:
reinforcement learning
Keywords:
Parallel Learning, Maximum Entropy, Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XZrVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper takes the parallel state-entropy objective of De Paola et al. (2025), where a population of policies maximizes the entropy of their combined state distribution, and scales it to large GPU simulators. K-Myriad represents the population as a multi-head policy with a shared trunk and trains it with a k-NN entropy estimate computed over the states visited by all heads. Experiments in Isaac Lab (Ant on several terrains, and in-hand cube manipulation) look at the entropy and diversity of the learned heads. They also test downstream sparse-reward tasks, where the head with the best initial success is fine-tuned with PPO. This initialization works better than random initialization and better than a single max-entropy policy, and Sec. 6.3 compares it with DIAYN and METRA.

Critical strengths

Making the parallel entropy objective work cheaply in realistic continuous-control simulators is useful, since before this it had only been shown in small discrete toy domains. The downstream gains over a single max-entropy policy are substantial on Cave and Pyramid (Fig. 4). For me this is the main thing in the paper's favor.

Critical weaknesses
  1. Since the objective comes from De Paola et al. and the method is mainly a scalable implementation, the paper needs to show that the collective-entropy objective is what helps downstream. I don't think it does yet. On the objective itself the population does worse than a single policy (Fig. 2), which the paper attributes to the smaller batch per head. Downstream, the population also gets to pick the best of 50 heads using the task reward, so in most tasks it starts fine-tuning well ahead of the single policy (Fig. 4). The random-initialization baseline accounts for selection among untrained networks, but not among trained exploratory policies. The obvious control would be the same 50 heads trained with uncoupled, per-head entropy (or independently trained max-entropy policies), followed by the same selection.

  2. The comparison with DIAYN and METRA (Sec. 6.3), the only one with established skill-discovery methods, does not look reliable. The DIAYN discriminator loss in Fig. 5c stays close to its chance-level starting value, so its skills are never really separated. That contradicts the text's explanation that DIAYN "forces extreme skill separation". And METRA reaches higher collective entropy than K-Myriad for most of pretraining (Fig. 5a). Both baselines lose their initial success early in fine-tuning, and the settings used to fine-tune them are not reported, so I can't tell whether this reflects the skills or the fine-tuning recipe.

Questions for authors
  1. Please add the uncoupled-heads control, with more seeds than the current four, and rerun Sec. 6.3 with baselines that actually learn, each fine-tuned with its own settings. If the coupled version still wins clearly, I would raise my score.
  2. Is the entropy computed on the full Ant observation or only on the x-y position? The baselines use x-y, which suits x-y goal reaching.
  3. How many runs are behind Fig. 4, and do the bands reflect seeds or goal locations? More generally, details like these, and the Sec. 6.3 setup, were hard to find or missing, which made the experiments harder to assess than necessary.
3 comments
AuthorsOpenReview ↗

Response to Reviewer XZrV (1/2)

We thank the reviewer to take the time to evaluate our work. We appreciate that our work demonstrate validity in his realistic application tackling complex continuos control task, throught a clear presentation. Based on the reviewer feedback we conducted an extensive campaign of additional experiments to demonstrate the clear validity of our approach.

W1,Q1 k-myriad vs Uncoupled heads

We thank the reviewer for this suggestion, which motivated a dedicated ablation.

To isolate the benefit of the collective objective itself, we introduce an ablation, Uncoupled-50, that is identical to K-Myriad in every respect except the coupling. It uses the same shared-trunk multi-head architecture, the same hyperparameters and learning rate, the same trajectory budget, and the same single-pass update. The only difference is in the k-NN entropy estimator. In K-Myriad, each state's neighbours are searched among the pooled states of all heads, so each head is rewarded for visiting regions that the other heads do not cover. In Uncoupled-50, neighbours are searched only among the head's own states, so each head maximizes its own state entropy independently. Any difference between the two methods is therefore caused by the collective objective, not by the architecture or the parallel implementation.

Pretraining (Fig. R1: Pretraining entropy, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO). On both locomotion domains, coupling yields a higher state entropy throughout training and in every 5 pretraining seed . On in-hand manipulation, Uncoupled-50 reaches a higher state entropy. Here the entropy is computed on the cube orientation, so it measures how widely orientations are spread, but not how steadily they are held.

To understand this difference, we decompose the population entropy into the contribution of the individual heads and the diversity between them. We measure between-head diversity as the Jensen–Shannon divergence among the heads' state distributions, normalized by its maximum value log⁡m\log m: D=H(dπp)−1m∑i=1mH(dπi)log⁡m∈[0,1],D = \frac{H\big(d_{\pi_p}\big) - \frac{1}{m}\sum_{i=1}^{m} H\big(d_{\pi_i}\big)}{\log m} \in [0, 1], where mm is the number of heads (m=50m = 50), estimated with the same k-NN estimator on equal-size samples (D=0D = 0: all heads visit the same states; D=1D = 1: disjoint regions).

Table 1: Between-head diversity D (mean ± 95% CI over 5 pretraining seeds)

EnvironmentK-MyriadUncoupled
Ant Empty0.43 ± 0.030.36 ± 0.02
Ant Maze0.35 ± 0.040.29 ± 0.01
Shadow Hand0.50 ± 0.080.37 ± 0.03

Table 1 shows that coupling works as intended in all three domains: K-Myriad's heads are more distinct from each other than Uncoupled's, in every seed. The domains differ in what this costs. On Ant, coupled heads individually cover as much as uncoupled ones and additionally spread apart, which increases the population entropy. On the hand, the entropy can also be raised individually, by each head rotating the cube more widely; coupling instead makes each head specialize to a narrower set of orientations. These narrower skills are markedly more stable: K-Myriad's heads keep the cube in hand about 40% longer before the first drop (Figure R2. https://imgur.com/a/ouOuNVO). Coupling thus trades individual orientation spread for distinct and stable manipulation skills.

Fine-tuning (Fig. R3: Fine-tuning, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO)

Table 2 Fine Tuning comparison for success rate curves

EnvironmentGoalsAUC K-MyriadAUC UncoupledBest-of-50 K-MyriadBest-of-50 Uncoupled
Ant Empty70.739 ± 0.0870.698 ± 0.1090.2860.256
Ant Maze90.352 ± 0.1760.227 ± 0.1650.1200.083
Shadow Hand110.146 ± 0.1110.141 ± 0.0960.201*0.208*

AUC: area under the fine-tuning success-rate curve. Best-of-50: success rate of the best of the 50 pretrained heads on each goal before fine-tuning

The pretraining advantage carries over downstream in locomotion. On Ant Empty, K-Myriad reaches a higher AUC than Uncoupled, and the gap widens on Ant Maze, where reaching the goals requires covering the whole maze. Consistently, the best of the 50 pretrained heads is already better before any fine-tuning: the coupled population contains policies that start closer to the downstream goals. On Shadow Hand, the two methods are on par over 11 target orientations. Since each coupled head covers fewer orientations than an uncoupled one, the advantage of more distinct heads does not translate into a downstream gain in this domain, while it yields a clear advantage in locomotion.

Summary. With architecture, compute and optimization held fixed, the collective objective alone makes the heads more distinct in all domains, which yields higher state entropy and better downstream performance in locomotion, and more stable skills in manipulation.

AuthorsOpenReview ↗

Response to Reviewer XZrV (2/2)

W2 Metra and Dyan clarification

To demonstrate the validity of our work, we follow the reviewer suggestion, testing also METRA and DIAYN in the other environments. We run the comparison of the end-to-end pipeline from the pretraining to the fine-tuning in the Ant Empty and Maze, as well as in the shadow hand environment. Since both of the methods do not maximize the entropy as their primary objective, to allow for a fair comparison we train them for the same number of epochs as K-Myriad (200 on Ant, 100 on Shadow Hand) and retain, for the downstream evaluation, the checkpoint with the highest state entropy. With a dedicated fine tuning of the hyper parameter of both the methods, the results are the following:

  • Figure R4: Pretraining entropy vs DIAYN and METRA.
  • Figure R5: Stability of the pretrained skills on Shadow Hand.
  • Figure R6: Downstream fine-tuning of K-Myriad, DIAYN and METRA.

K-myriad vs DIAYN vs METRA (https://imgur.com/a/W8cJVez)

We can summarize the figures with:

  1. K-Myriad clearly outperforms DIAYN in fine-tuning across all environments, and in pretraining entropy on both locomotion domains.
  2. METRA reaches comparable or higher state entropy during pretraining, which translates into higher fine-tuning performance only on Ant Maze, where its temporal-distance objective is well aligned with the locomotion task.

Q2 Entropy calculation clarification

The state entropy calculation is different among the environments:

  • Ant empty, entropy is calculated over the X-Y coordinate of the ant torso along the trajectory visited
  • Ant maze, equal to the emtpy environement
  • Ant cave and pyramid, the z value, or namely the height of the torso is considered into the calculation
  • Inhand env, calculates the entropy based on the orientation of the cube.

each baseline shares the same entropy calculation. The input for the policy is the entire state space, 62-D and 149-D for the ant and the hand respectively. We apologize if this aspect was not clear from the main paper. It will be updated in the camera ready version.

Q3 Fine-tuning experiment's details

The fine tuning experiments are validated over goals, not over seeds. All the details about the experiments are listed in the table 3 and 4 in the appendix section D. Additionally we will clarify the goal evaluation, updating the caption of the figure 4.

Dear Reviewer XZrV,

Thank you again for your constructive review and concrete experimental suggestions. We wanted to kindly follow up to ensure our responses,particularly the Uncoupled-50 control (Figs. R1–R3, Tables 1–2), the expanded and tuned DIAYN/METRA comparisons across all environments (Figs. R4–R6), and the clarifications on state subspaces and Fig. 4,have addressed the conditions you outlined for reconsidering your score.

Please let us know if any additional details would be helpful before the discussion period ends.

Best regards,
The Authors

Reviewer_khW6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

K-Myriad targets a key flaw of traditional parallel RL: it optimizes only a single policy and neglects exploration diversity. Using a shared backbone with multiple independent heads, it trains up to 50 heterogeneous policies synchronously on a single GPU. Its core objective is to maximize the collective state visitation entropy across policies, with end-to-end gradients via a KNN non-parametric entropy estimator, driving policies to explore differentiated regions and achieving efficient population-level full-domain exploration. Experiments on Isaac Sim quadruped locomotion and Shadow Hand manipulation show significant gains in sample efficiency and final performance over random initialization and single-agent maximum-entropy pre-training. Results further demonstrate that K-Myriad learns a broad set of distinct policies, highlighting its effectiveness for collective exploration and paving the way for novel parallelization strategies.

Critical strengths

None

Critical weaknesses

1.The K-Myriad framework relies heavily on existing mature solutions. The core objective of parallel state entropy maximization was proposed by De Paola et al. The shared backbone plus multiple independent heads architecture is a common design in multi-task learning. The gradient computation of k-NN entropy estimation follows the classic paradigm of prior unsupervised exploration work. The paper does not propose breakthrough improvements in basic theory or algorithmic logic. It is essentially an engineering adaptation of existing parallel entropy maximization methods, with insufficient novelty.

2.The experimental design has limitations, and the conclusions are not sufficiently convincing. The experiments clearly show that the 50-agent configuration converges significantly slower than the single-agent one, yet this is attributed only to batch size differences, without proposing any algorithmic-level optimization solution.

Questions for authors

1.The K-Myriad framework relies heavily on existing mature solutions. It follows the paradigms of multi-task learning and classic k-NN work, without theoretical or algorithmic innovation. 2.The trend in the Figure 2 curves shows that as the amount of agent data increases, overall convergence slows down. In simple blank environments and maze environments, the performance of 10-agent and 50-agent populations rises noticeably more slowly than that of a single agent. How can convergence speed and convergence be guaranteed with multiple agents? 3.The shared backbone receives gradients from multiple policy heads. If the differences are too large, how is convergence guaranteed? Or will there be gradient cancellation? 4.The core advantage of K-Myriad is that agents in parallel master different policies. Selecting only a single optimal policy for initialization is equivalent to directly abandoning the skills mastered by the other policies—isn't that somewhat wasteful? 5.The comparison with unsupervised skill discovery methods includes only 2018 DIAYN and 2021 Metra. Please supplement with more recent methods under fair data conditions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer khW6 (1/3)

We would like to thank the reviewer for their comments. We are committed to the robustness of the experimental designg and, following the reviewer's comments, we conducted an extensive experimental campaign to further support the validity of our approach.

W1,Q1 Novelty

We agree that the parallel maximum-state-entropy objective was introduced in prior work (De Paola et al., 2025). However, making an established objective practical at scale is a recognized contribution in our field. The maximum state entropy objective was formulated by Hazan et al. (2019), and the algorithms that made it work in continuous control (MEPOL, Mutti et al., 2021; APT, Liu & Abbeel, 2021) are considered contributions in their own right. Similarly, recent work at top venues has shown the value of understanding and enabling deep RL at scale, from the effect of massively parallel on-policy data collection [1] to large-scale RL for dexterous manipulation [2] and scaling network architectures [3].

K-Myriad provides this step for the parallel maximum-state-entropy objective. It brings the objective from discrete, low-dimensional domains to 62-D and 149-D continuous simulators through a coupled k-NN estimator and a single-pass multi-head update.

[1] Mayor et al., "The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks", ICML 2025. [2] Yin et al., "Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning", ICLR 2026. [3] Lee et al., "SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning", ICLR 2025.

Q3 Gradients cancellation in shared trunks

We thank the reviewer for this question. Unlike multi-task learning, where each head optimizes its own loss and their gradients can compete on the shared backbone, K-Myriad optimizes a single scalar objective: the k-NN entropy of the pooled state distribution of all agents. The per-head terms are not separate objectives but additive components of one estimator. Hence the backbone receives exactly the gradient of this joint objective with respect to the shared parameters, and any partial cancellation between heads' contributions is itself part of the correct ascent direction: it indicates that the shared features should not change along that direction, while each head can still specialize through its own parameters. Empirically, state entropy increases steadily and plateaus in every pretraining seed, in all environments, with low variance across seeds.

Q4 Fine-tuning method clarification

The ultimate goal of our approach is to learn diverse covering policies to initialize a fine-tuning process. However, K-Myriad is a pre-training method rather than a downstream RL algorithm; specialized fine-tuning strategies are therefore outside the scope of this work. Selecting a single head for fine-tuning serves to demonstrate the quality of the pre-training. Let's consider this assumption through a geometric interpretation. Consider the state occupancy polytope (see Eysenbach et al 2021 "The information geometry of unsupervised reinforcement learning"). The optimal policy for a RL task will be on one of the vertices of the polytope. Now consider 50 policies laying down in the same polytope. If those policies achieve coverage and diversity, the chances of having one policy close to the optimal policy for a given task is high. Instead, 50 identical policies can be close to some vertices of the polytope, but they will be far from others. Thus, the best-of-50 will be good for some tasks and bad for others. Instead, the best-of-50 of our diverse policies tends to fall closer to the optimal policy. This guide us in the protocol evaluation used in the main paper.

AuthorsOpenReview ↗

Response to Reviewer khW6 (2/3)

W2,Q2 Empirical superiority

To isolate the benefit of the collective objective itself, we introduce an ablation, Uncoupled-50, that is identical to K-Myriad in every respect except the coupling. It uses the same shared-trunk multi-head architecture, the same hyperparameters and learning rate, the same trajectory budget, and the same single-pass update. The only difference is in the k-NN entropy estimator. In K-Myriad, each state's neighbours are searched among the pooled states of all heads, so each head is rewarded for visiting regions that the other heads do not cover. In Uncoupled-50, neighbours are searched only among the head's own states, so each head maximizes its own state entropy independently. Any difference between the two methods is therefore caused by the collective objective, not by the architecture or the parallel implementation.

Pretraining (Fig. R1: Pretraining entropy, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO). On both locomotion domains, coupling yields a higher state entropy throughout training and in every 5 pretraining seed . On in-hand manipulation, Uncoupled-50 reaches a higher state entropy . Here the entropy is computed on the cube orientation, so it measures how widely orientations are spread, but not how steadily they are held.

To understand this difference, we decompose the population entropy into the contribution of the individual heads and the diversity between them. We measure between-head diversity as the Jensen–Shannon divergence among the heads' state distributions, normalized by its maximum value log⁡m\log m: D=H(dπp)−1m∑i=1mH(dπi)log⁡m∈[0,1],D = \frac{H\big(d_{\pi_p}\big) - \frac{1}{m}\sum_{i=1}^{m} H\big(d_{\pi_i}\big)}{\log m} \in [0, 1], where mm is the number of heads (m=50m = 50), estimated with the same k-NN estimator on equal-size samples (D=0D = 0: all heads visit the same states; D=1D = 1: disjoint regions).

Table 1: Between-head diversity D (mean ± 95% CI over 5 pretraining seeds)

EnvironmentK-MyriadUncoupled
Ant Empty0.43 ± 0.030.36 ± 0.02
Ant Maze0.35 ± 0.040.29 ± 0.01
Shadow Hand0.50 ± 0.080.37 ± 0.03

Table 1 shows that coupling works as intended in all three domains: K-Myriad's heads are more distinct from each other than Uncoupled's, in every seed. The domains differ in what this costs. On Ant, coupled heads individually cover as much as uncoupled ones and additionally spread apart, which increases the population entropy. On the hand, the entropy can also be raised individually, by each head rotating the cube more widely; coupling instead makes each head specialize to a narrower set of orientations. These narrower skills are markedly more stable: K-Myriad's heads keep the cube in hand about 40% longer before the first drop (Figure R2. https://imgur.com/a/ouOuNVO). Coupling thus trades individual orientation spread for distinct and stable manipulation skills.

Fine-tuning (Fig. R3: Fine-tuning, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO)

Table 2 Fine Tuning comparison for success rate curves

EnvironmentGoalsAUC K-MyriadAUC UncoupledBest-of-50 K-MyriadBest-of-50 Uncoupled
Ant Empty70.739 ± 0.0870.698 ± 0.1090.2860.256
Ant Maze90.352 ± 0.1760.227 ± 0.1650.1200.083
Shadow Hand110.146 ± 0.1110.141 ± 0.0960.201*0.208*

AUC: area under the fine-tuning success-rate curve. Best-of-50: success rate of the best of the 50 pretrained heads on each goal before fine-tuning

The pretraining advantage carries over downstream in locomotion. On Ant Empty, K-Myriad reaches a higher AUC than Uncoupled ,and the gap widens on Ant Maze , where reaching the goals requires covering the whole maze. Consistently, the best of the 50 pretrained heads is already better before any fine-tuning: the coupled population contains policies that start closer to the downstream goals. On Shadow Hand, the two methods are on par over 11 target orientations. Since each coupled head covers fewer orientations than an uncoupled one, the advantage of more distinct heads does not translate into a downstream gain in this domain, while it yields a clear advantage in locomotion.

Summary. With architecture, compute and optimization held fixed, the collective objective alone makes the heads more distinct in all domains, which yields higher state entropy and better downstream performance in locomotion, and more stable skills in manipulation.

Thanks for the detailed responses and additional experiments. I find the Uncoupled-50 ablation useful and appreciate the expanded comparisons with DIAYN and METRA. These additions address several of my concerns and make the empirical results more convincing, although some questions regarding methodological novelty remain. Overall, I appreciate the authors' efforts and am willing to raise my score from 1 to 2.

Reviewer_SM3cOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper, of the so-called K-Myriad, basically is maximizing the collective state entropy induced by a population of parallel polices. The is built upon a cited paper, De Paola (2025). Different to the existing work, this paper however, uses a parallel state-entropy maximization in continuous domains. The authors also applied a shared trunk with separate policy heads, to pre-train a collection of policies while interacting with multiple of independent environment replicas. The policies are jointly maximized the entropy of their averaged state-visitation distribution. In this stage, the learning specified exploration strategies without task rewards. This seems to be in a similar spirit of the earlier internal curiosity rewards.

To jump-start the downstream RL, the paper evaluates each pre-trained policy using task rewards, select the policy with the highest initial performance, and uses it to initialized PPO fine-tuning.

The authors have included some experiments for high-dimensional continuous-control tasks, such as Ant locomotion across multiple environment variations etc.

Critical strengths

good engineering design, where the shared-network, with the trunk and multiple heads implementation supports large policy populations. The reported computations savings compare with that of the separate networks.

The reported comparisons shows empirical evidence of benefit over single policy pre-training on selecting among multiple random initializations.

Critical weaknesses

I don't find this paper to have a significant novelty. The collective entropy objective follows that De Paola (ICML 2025) paper. The main advance appears to be just the scalable implementation and downstream tasks.

The actual isolated contributions from the collective objectives itself is not clearly demonstrated.

DIAYN and METRA comparisons are restricted to Ant Empty, so the natural question to raise is if such performance can be replicate d in other environment.

Only PPO is used, for other RL optimization techniques such as GRPO should also be at least discussed.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to reviewer SM3c (1/2)

We thank the reviewer to take time to review our work, and we especially appreciate that our paper is recognized to show the empirical benefit that our method take over the single policy pre-training, through our clear presentation and soundness. We conducted an extensive rebuttal campaign of experiments to takle specifically each weakness highlighted.

W1 Novelty

We agree that the parallel maximum-state-entropy objective was introduced in prior work (De Paola et al., 2025). However, making an established objective practical at scale is a recognized contribution in our field. The maximum state entropy objective was formulated by Hazan et al. (2019), and the algorithms that made it work in continuous control (MEPOL, Mutti et al., 2021; APT, Liu & Abbeel, 2021) are considered contributions in their own right. Similarly, recent work at top venues has shown the value of understanding and enabling deep RL at scale, from the effect of massively parallel on-policy data collection [1] to large-scale RL for dexterous manipulation [2] and scaling network architectures [3].

K-Myriad provides this step for the parallel maximum-state-entropy objective. It brings the objective from discrete, low-dimensional domains to 62-D and 149-D continuous simulators through a coupled k-NN estimator and a single-pass multi-head update.

[1] Mayor et al., "The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks", ICML 2025. [2] Yin et al., "Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning", ICLR 2026. [3] Lee et al., "SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning", ICLR 2025.

W3 DIAYN and METRA experiments extension

To demonstrate the validity of our work, we follow the reviewer suggestion, testing also METRA and DIAYN in the other environments. We run the comparison of the end-to-end pipeline from the pretraining to the fine-tuning in the Ant Empty and Maze, as well as in the shadow hand environment. Since both of the methods do not maximize the entropy as their primary objective, to allow a fair comparison we train them for the same number of epochs as K-Myriad (200 on Ant, 100 on Shadow Hand) and retain, for the downstream evaluation, the checkpoint with the highest state entropy. With a dedicated fine tuning of the hyper parameter of both the methods, the results are the following:

  • Figure R4: Pretraining entropy vs DIAYN and METRA.
  • Figure R5: Stability of the pretrained skills on Shadow Hand.
  • Figure R6: Downstream fine-tuning of K-Myriad, DIAYN and METRA.

K-myriad vs DIAYN vs METRA (https://imgur.com/a/W8cJVez)

We can summarize the figures with:

  1. K-Myriad clearly outperforms DIAYN in fine-tuning across all environments, and in pretraining entropy on both locomotion domains.
  2. METRA reaches comparable or higher state entropy during pretraining, which translates into higher fine-tuning performance only on Ant Maze, where its temporal-distance objective is well aligned with the locomotion task.

W4 GRPO as fine-tuning method

We thank the reviewer for the suggestion. K-Myriad is a pretraining method: its output is a policy initialization, and the fine-tuning optimizer is orthogonal to our contribution. We used PPO for all methods, with identical hyperparameters, to isolate the effect of pretraining. PPO is also the standard on-policy algorithm for massively parallel GPU simulation (Isaac Lab). GRPO was designed for language-model fine-tuning: it normalizes terminal rewards within a group of completions sampled from the same prompt, and assigns the resulting advantage uniformly to all tokens, without a critic. Our tasks are long-horizon continuous-control problems, where per-timestep credit assignment through a learned value function is important and repeated rollouts from an identical state are not available.

AuthorsOpenReview ↗

Response to reviewer SM3c (2/2)

W2 Advantages of the collective objective

To isolate the benefit of the collective objective itself, we introduce an ablation, Uncoupled-50, that is identical to K-Myriad in every respect except the coupling. It uses the same shared-trunk multi-head architecture, the same hyperparameters and learning rate, the same trajectory budget, and the same single-pass update. The only difference is in the k-NN entropy estimator. In K-Myriad, each state's neighbours are searched among the pooled states of all heads, so each head is rewarded for visiting regions that the other heads do not cover. In Uncoupled-50, neighbours are searched only among the head's own states, so each head maximizes its own state entropy independently. Any difference between the two methods is therefore caused by the collective objective, not by the architecture or the parallel implementation.

Pretraining (Fig. R1: Pretraining entropy, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO). On both locomotion domains, coupling yields a higher state entropy throughout training and in every 5 pretraining seed . On in-hand manipulation, Uncoupled-50 reaches a higher state entropy. Here the entropy is computed on the cube orientation, so it measures how widely orientations are spread, but not how steadily they are held.

To understand this difference, we decompose the population entropy into the contribution of the individual heads and the diversity between them. We measure between-head diversity as the Jensen–Shannon divergence among the heads' state distributions, normalized by its maximum value log⁡m\log m: D=H(dπp)−1m∑i=1mH(dπi)log⁡m∈[0,1],D = \frac{H\big(d_{\pi_p}\big) - \frac{1}{m}\sum_{i=1}^{m} H\big(d_{\pi_i}\big)}{\log m} \in [0, 1], where mm is the number of heads (m=50m = 50), estimated with the same k-NN estimator on equal-size samples (D=0D = 0: all heads visit the same states; D=1D = 1: disjoint regions).

Table 1: Between-head diversity D (mean ± 95% CI over 5 pretraining seeds)

EnvironmentK-MyriadUncoupled
Ant Empty0.43 ± 0.030.36 ± 0.02
Ant Maze0.35 ± 0.040.29 ± 0.01
Shadow Hand0.50 ± 0.080.37 ± 0.03

Table 1 shows that coupling works as intended in all three domains: K-Myriad's heads are more distinct from each other than Uncoupled's, in every seed. The domains differ in what this costs. On Ant, coupled heads individually cover as much as uncoupled ones and additionally spread apart, which increases the population entropy. On the hand, the entropy can also be raised individually, by each head rotating the cube more widely; coupling instead makes each head specialize to a narrower set of orientations. These narrower skills are markedly more stable: K-Myriad's heads keep the cube in hand about 40% longer before the first drop( Figure R2. https://imgur.com/a/ouOuNVO). Coupling thus trades individual orientation spread for distinct and stable manipulation skills.

Fine-tuning (Fig. R3: Fine-tuning, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO)

Table 2 Fine Tuning comparison for success rate curves

EnvironmentGoalsAUC K-MyriadAUC UncoupledBest-of-50 K-MyriadBest-of-50 Uncoupled
Ant Empty70.739 ± 0.0870.698 ± 0.1090.2860.256
Ant Maze90.352 ± 0.1760.227 ± 0.1650.1200.083
Shadow Hand110.146 ± 0.1110.141 ± 0.0960.201*0.208*

AUC: area under the fine-tuning success-rate curve. Best-of-50: success rate of the best of the 50 pretrained heads on each goal before fine-tuning

The pretraining advantage carries over downstream in locomotion. On Ant Empty, K-Myriad reaches a higher AUC than Uncoupled, and the gap widens on Ant Maze , where reaching the goals requires covering the whole maze. Consistently, the best of the 50 pretrained heads is already better before any fine-tuning: the coupled population contains policies that start closer to the downstream goals. On Shadow Hand, the two methods are on par over 11 target orientations. Since each coupled head covers fewer orientations than an uncoupled one, the advantage of more distinct heads does not translate into a downstream gain in this domain, while it yields a clear advantage in locomotion.

Summary. With architecture, compute and optimization held fixed, the collective objective alone makes the heads more distinct in all domains, which yields higher state entropy and better downstream performance in locomotion, and more stable skills in manipulation.

Dear Reviewer SM3c,

Thank you again for your review and time. We wanted to kindly follow up on our responses above, where we added the Uncoupled-50 ablation isolating the collective objective (Figs. R1–R3, Tables 1–2), extended the DIAYN and METRA baselines across all domains (Figs. R4–R6), and clarified the fine-tuning optimizer choice. Please let us know if these additions address your concerns or if you have any remaining questions for discussion.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer khW6 (3/3)

Q5 Baseline Validity

We respectfully ask the reviewer to reconsider DIAYN and METRA validity as correct baselines to demonstrate the validity of k-myriad at the intersection of skill discovery and pretraining via entropy maximization. Metra (ICLR'24) is, to the best of our knowledge, the SOTA for pretraining skills in the mutual information framework.

We conduct further experiments to robustify the comparison with the baselines with the following protocol. Since both of the methods do not maximize the entropy as their primary objective, to allow for a fair comparison we train them for the same number of epochs as K-Myriad (200 on Ant, 100 on Shadow Hand) and retain, for the downstream evaluation, the checkpoint with the highest state entropy. With a dedicated fine tuning of the hyper parameter of both the methods, the results are the following:

  • Figure R4: Pretraining entropy vs DIAYN and METRA.
  • Figure R5: Stability of the pretrained skills on Shadow Hand.
  • Figure R6: Downstream fine-tuning of K-Myriad, DIAYN and METRA.

K-myriad vs DIAYN vs METRA (https://imgur.com/a/W8cJVez)

We can summarize the figures with:

  1. K-Myriad clearly outperforms DIAYN in fine-tuning across all environments, and in pretraining entropy on both locomotion domains.
  2. METRA reaches comparable or higher state entropy during pretraining, which translates into higher fine-tuning performance only on Ant Maze, where its temporal-distance objective is well aligned with the locomotion task.
AuthorsOpenReview ↗

General Response to Reviews

Dear Reviewers and Area Chair,

Thank you again for your time and feedback. As the discussion phase progresses, we wanted to provide a brief, unified summary of the new experimental controls we ran to directly address the core questions raised across the reviews:

  1. Isolating the collective objective (Uncoupled-50 ablation): Holding the shared-trunk multi-head architecture, trajectory budget, and best-of-50 selection identical, coupling the kk-NN estimator increases between-head diversity across every domain (Table 1). This translates directly into higher pretraining entropy and stronger downstream fine-tuning AUC on Ant Empty and Ant Maze , as well asmore stable object retention on Shadow Hand (Figs. R1–R3).
  2. Expanded, hyperparameter-tuned skill-discovery baselines across all domains: We extended the DIAYN and METRA evaluations with dedicated tuning to Ant Empty, Ant Maze, and Shadow Hand (Figs. R4–R6). K-Myriad outperforms DIAYN across all environments and outperforms or matches METRA on Ant Empty and Shadow Hand, while METRA's temporal-distance bias only provides an edge on Ant Maze.
  3. Experimental clarifications: We clarified the exact state subspaces used for entropy estimation across environments and the goal-level validation protocol in Fig. 4, which will be incorporated into the revision.

Figures R1-R3: https://imgur.com/a/ouOuNVO

Figures R4-R6: https://imgur.com/a/W8cJVez

We hope these controls address your remaining questions, and we would be grateful to know if any further details or clarifications would be helpful before the discussion period closes, to revise your rating on our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.