- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper takes the parallel state-entropy objective of De Paola et al. (2025), where a population of policies maximizes the entropy of their combined state distribution, and scales it to large GPU simulators. K-Myriad represents the population as a multi-head policy with a shared trunk and trains it with a k-NN entropy estimate computed over the states visited by all heads. Experiments in Isaac Lab (Ant on several terrains, and in-hand cube manipulation) look at the entropy and diversity of the learned heads. They also test downstream sparse-reward tasks, where the head with the best initial success is fine-tuned with PPO. This initialization works better than random initialization and better than a single max-entropy policy, and Sec. 6.3 compares it with DIAYN and METRA.
Critical strengths
Making the parallel entropy objective work cheaply in realistic continuous-control simulators is useful, since before this it had only been shown in small discrete toy domains. The downstream gains over a single max-entropy policy are substantial on Cave and Pyramid (Fig. 4). For me this is the main thing in the paper's favor.
Critical weaknesses
-
Since the objective comes from De Paola et al. and the method is mainly a scalable implementation, the paper needs to show that the collective-entropy objective is what helps downstream. I don't think it does yet. On the objective itself the population does worse than a single policy (Fig. 2), which the paper attributes to the smaller batch per head. Downstream, the population also gets to pick the best of 50 heads using the task reward, so in most tasks it starts fine-tuning well ahead of the single policy (Fig. 4). The random-initialization baseline accounts for selection among untrained networks, but not among trained exploratory policies. The obvious control would be the same 50 heads trained with uncoupled, per-head entropy (or independently trained max-entropy policies), followed by the same selection.
-
The comparison with DIAYN and METRA (Sec. 6.3), the only one with established skill-discovery methods, does not look reliable. The DIAYN discriminator loss in Fig. 5c stays close to its chance-level starting value, so its skills are never really separated. That contradicts the text's explanation that DIAYN "forces extreme skill separation". And METRA reaches higher collective entropy than K-Myriad for most of pretraining (Fig. 5a). Both baselines lose their initial success early in fine-tuning, and the settings used to fine-tune them are not reported, so I can't tell whether this reflects the skills or the fine-tuning recipe.
Questions for authors
- Please add the uncoupled-heads control, with more seeds than the current four, and rerun Sec. 6.3 with baselines that actually learn, each fine-tuned with its own settings. If the coupled version still wins clearly, I would raise my score.
- Is the entropy computed on the full Ant observation or only on the x-y position? The baselines use x-y, which suits x-y goal reaching.
- How many runs are behind Fig. 4, and do the bands reflect seeds or goal locations? More generally, details like these, and the Sec. 6.3 setup, were hard to find or missing, which made the experiments harder to assess than necessary.
3 comments
Response to Reviewer XZrV (2/2)
W2 Metra and Dyan clarification
To demonstrate the validity of our work, we follow the reviewer suggestion, testing also METRA and DIAYN in the other environments. We run the comparison of the end-to-end pipeline from the pretraining to the fine-tuning in the Ant Empty and Maze, as well as in the shadow hand environment. Since both of the methods do not maximize the entropy as their primary objective, to allow for a fair comparison we train them for the same number of epochs as K-Myriad (200 on Ant, 100 on Shadow Hand) and retain, for the downstream evaluation, the checkpoint with the highest state entropy. With a dedicated fine tuning of the hyper parameter of both the methods, the results are the following:
- Figure R4: Pretraining entropy vs DIAYN and METRA.
- Figure R5: Stability of the pretrained skills on Shadow Hand.
- Figure R6: Downstream fine-tuning of K-Myriad, DIAYN and METRA.
K-myriad vs DIAYN vs METRA (https://imgur.com/a/W8cJVez)
We can summarize the figures with:
- K-Myriad clearly outperforms DIAYN in fine-tuning across all environments, and in pretraining entropy on both locomotion domains.
- METRA reaches comparable or higher state entropy during pretraining, which translates into higher fine-tuning performance only on Ant Maze, where its temporal-distance objective is well aligned with the locomotion task.
Q2 Entropy calculation clarification
The state entropy calculation is different among the environments:
- Ant empty, entropy is calculated over the X-Y coordinate of the ant torso along the trajectory visited
- Ant maze, equal to the emtpy environement
- Ant cave and pyramid, the z value, or namely the height of the torso is considered into the calculation
- Inhand env, calculates the entropy based on the orientation of the cube.
each baseline shares the same entropy calculation. The input for the policy is the entire state space, 62-D and 149-D for the ant and the hand respectively. We apologize if this aspect was not clear from the main paper. It will be updated in the camera ready version.
Q3 Fine-tuning experiment's details
The fine tuning experiments are validated over goals, not over seeds. All the details about the experiments are listed in the table 3 and 4 in the appendix section D. Additionally we will clarify the goal evaluation, updating the caption of the figure 4.
Follow up
Dear Reviewer XZrV,
Thank you again for your constructive review and concrete experimental suggestions. We wanted to kindly follow up to ensure our responses,particularly the Uncoupled-50 control (Figs. R1–R3, Tables 1–2), the expanded and tuned DIAYN/METRA comparisons across all environments (Figs. R4–R6), and the clarifications on state subspaces and Fig. 4,have addressed the conditions you outlined for reconsidering your score.
Please let us know if any additional details would be helpful before the discussion period ends.
Best regards,
The Authors
Response to Reviewer XZrV (1/2)
We thank the reviewer to take the time to evaluate our work. We appreciate that our work demonstrate validity in his realistic application tackling complex continuos control task, throught a clear presentation. Based on the reviewer feedback we conducted an extensive campaign of additional experiments to demonstrate the clear validity of our approach.
W1,Q1 k-myriad vs Uncoupled heads
We thank the reviewer for this suggestion, which motivated a dedicated ablation.
To isolate the benefit of the collective objective itself, we introduce an ablation, Uncoupled-50, that is identical to K-Myriad in every respect except the coupling. It uses the same shared-trunk multi-head architecture, the same hyperparameters and learning rate, the same trajectory budget, and the same single-pass update. The only difference is in the k-NN entropy estimator. In K-Myriad, each state's neighbours are searched among the pooled states of all heads, so each head is rewarded for visiting regions that the other heads do not cover. In Uncoupled-50, neighbours are searched only among the head's own states, so each head maximizes its own state entropy independently. Any difference between the two methods is therefore caused by the collective objective, not by the architecture or the parallel implementation.
Pretraining (Fig. R1: Pretraining entropy, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO). On both locomotion domains, coupling yields a higher state entropy throughout training and in every 5 pretraining seed . On in-hand manipulation, Uncoupled-50 reaches a higher state entropy. Here the entropy is computed on the cube orientation, so it measures how widely orientations are spread, but not how steadily they are held.
To understand this difference, we decompose the population entropy into the contribution of the individual heads and the diversity between them. We measure between-head diversity as the Jensen–Shannon divergence among the heads' state distributions, normalized by its maximum value logm: D=logmH(dπp)−m1∑i=1mH(dπi)∈[0,1], where m is the number of heads (m=50), estimated with the same k-NN estimator on equal-size samples (D=0: all heads visit the same states; D=1: disjoint regions).
Table 1: Between-head diversity D (mean ± 95% CI over 5 pretraining seeds)
Table 1 shows that coupling works as intended in all three domains: K-Myriad's heads are more distinct from each other than Uncoupled's, in every seed. The domains differ in what this costs. On Ant, coupled heads individually cover as much as uncoupled ones and additionally spread apart, which increases the population entropy. On the hand, the entropy can also be raised individually, by each head rotating the cube more widely; coupling instead makes each head specialize to a narrower set of orientations. These narrower skills are markedly more stable: K-Myriad's heads keep the cube in hand about 40% longer before the first drop (Figure R2. https://imgur.com/a/ouOuNVO). Coupling thus trades individual orientation spread for distinct and stable manipulation skills.
Fine-tuning (Fig. R3: Fine-tuning, K-Myriad vs Uncoupled https://imgur.com/a/ouOuNVO)
Table 2 Fine Tuning comparison for success rate curves
AUC: area under the fine-tuning success-rate curve. Best-of-50: success rate of the best of the 50 pretrained heads on each goal before fine-tuning
The pretraining advantage carries over downstream in locomotion. On Ant Empty, K-Myriad reaches a higher AUC than Uncoupled, and the gap widens on Ant Maze, where reaching the goals requires covering the whole maze. Consistently, the best of the 50 pretrained heads is already better before any fine-tuning: the coupled population contains policies that start closer to the downstream goals. On Shadow Hand, the two methods are on par over 11 target orientations. Since each coupled head covers fewer orientations than an uncoupled one, the advantage of more distinct heads does not translate into a downstream gain in this domain, while it yields a clear advantage in locomotion.
Summary. With architecture, compute and optimization held fixed, the collective objective alone makes the heads more distinct in all domains, which yields higher state entropy and better downstream performance in locomotion, and more stable skills in manipulation.