- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces the maximize mixture, minimize components entropy objective for learning a fixed bank of K state-dependent, one-step motor primitives without task reward, which a downstream discrete controller can then select from. The policy is a state-conditioned Gaussian mixture trained to maximize the cumulative entropy of the mixture while penalizing the entropy of each component ("maxi-mix-mini-com").
For the tabular case, the authors derive alternating closed-form updates for components, weights and responsibilities with monotonic value improvement. For continuous control, the mixture is trained with a SAC-style off-policy algorithm. The component means are then frozen and used as K discrete actions for a discrete PPO controller. Experiments on MuJoCo and other benchmarks compare against continuous PPO, random action sets and unsupervised skill discovery (DIAYN, DADS, LSD, METRA).
Critical strengths
The problem framing is clear and well motivated: a reward-free, reusable, one-step action vocabulary that separates motor learning from task decisions. The evaluation covers many environments and task transfer.
Critical weaknesses
-
Missing discretization baselines. The paper positions itself as online action quantization, and its main claim is that a discrete controller over learned tokens retains continuous-control performance (Table 3). The work cites other methods such as Tang & Agrawal 202, Seyde 2021 or Seo 2025, but does not provide comparisons. The only discretaziation baseline is a random action set.
Offline discretization methods such as GMM fits on the reward-free explorer are not tested, which would test if learning the action vocabulary online matters.
-
Skill discovery baselines. The results in Tab. 1 are the only quantitative experiments comparing against other skill-discovery methods. From my limited understanding, these methods are optimized for multi step "action chunks", and not suited for single step execution. The ablations in A.7 show, that the proposed method performance halves when holding the same action for 5 steps. Thus, these results could be confounded by the effect of the time scale on the policies.
Questions for authors
How does the frozen token bank compare to an established online discretization baseline, e.g. PPO with a factorized per-dimension categorical policy (Tang & Agrawal, 2020), under the same training budget? If the learned K=16/64 tokens match or outperform this baseline, particularly on Humanoid and Walker2d, this would substantially strengthen the claim that learning the token bank is beneficial.
6 comments
Response 2/2
Offline GMM discretization baseline
As discussed in Appendix A.1 (lines 723–752), offline fitting begins with a fixed dataset and compresses its existing action support by selecting a low number of representative actions. Once the data are supplied, conditional-GMM fitting is a stationary supervised problem that does not face the exploration problem present in our problem and solution. Indeed, our reward-free online learner must instead generate its own experience while jointly maintaining broad mixture-level action coverage and specialized component identities; consequently, the learned bank and its visitation distribution co-evolve. This is why we treated offline methods as a complementary setting in Related Work.
Following the reviewer’s suggestion again, we ran a matched post-hoc conditional-GMM baseline on Ant-v5. For each of five seeds, our maximum-entropy explorer collected transitions. Using the same state-conditioned architecture and update count as our learner, we fitted tanh-GMMs with offline by conditional negative log-likelihood, retained their deterministic component modes as frozen tokens, and trained the same Table 3 discrete PPO controller for steps.
| Bank | K=16 | K=64 |
|---|---|---|
| Offline conditional GMM (n=5) | 1505 ± 195 | 1567 ± 238 |
| Ours, Table 3 (n=10) | 2937 ± 23 | 3581 ± 32 |
Values are mean return 95% CI.
Our bank achieves around 2x the GMM return for and , respectively. All GMM runs had plateaued by steps, with the best evaluation improving by at most 88 thereafter, and increasing produced no clear gain. Conditional likelihood need only fit the overall ; it does not require every component identity to become distinct and useful after its weights and variances are discarded. Our objective explicitly encourages this specialization.
These results indicate that our joint entropy objective yields substantially more useful frozen behavioral tokens than offline conditional maximum-likelihood fitting. In particular, likelihood fitting does not explicitly enforce the component specialization encouraged by our objective.
We will include these results and clarify the distinction between online and offline token learning in the revised manuscript.
Fair temporal horizon for skill-discovery baselines
We agree with the reviewer that DIAYN, DADS, LSD, and METRA are primarily designed to discover temporally extended skills, and this was discussed in our manuscript. Our experiment addresses a different question: Can conventional skill-discovery methods serve as one-step behavioral tokens? To evaluate this, all methods use a discrete PPO controller selecting among 64 frozen tokens or skills at every environment step. Importantly, one-step selection does not prevent temporally extended execution: the controller can, in principle, reproduce a longer skill by repeatedly selecting the same component for 10, 25, or more consecutive steps, but this is rarely done by our learned high level controllers. What our results show is that having one-step composability of tokens adds additional flexibility to better solve downstream tasks that skills do not have.
We will further clarify these crucial distinctions in the revised manuscript.
Temporal Horizon
I thank the authors for their detailed responses to all reviews. In particular, the additional experiments are highly appreciated and address most of my previous concerns.
I still have some questions regarding Tab. 1 and the comparison against unsupervised skill-discovery methods. I may be misunderstanding some aspects of the baseline methods or experimental setup, so I would appreciate some clarification.
My understanding is that the USD baselines are designed to discover temporally extended skills, which I imagine as sequences of actions (e.g., ). I am unsure how these are converted into the 64 frozen choices used by the downstream controller. If selecting the same skill repeatedly simply executes its initial action again, the temporal structure of the original skill would be lost.
Since the USD methods are designed around temporally extended behaviors, I am not convinced that evaluating them as one-step tokens provides a meaningful comparison of their discovery capabilities. I therefore think that the conclusions should be limited to the particular one-step execution setting tested here.
Response 1/2
We thank the reviewer for the efforts and the high quality review of our paper.
We thank the reviewer for suggesting this additional comparison. We would like to clarify first that our objective is to learn a compact, reward-free, state-dependent action vocabulary that can be frozen and reused across downstream tasks while retaining competitive control performance. Factorized categorical PPO (ref: Tang and Agrawal (2020)) addresses a different problem: it factorizes the policy into per-dimension components, and discretizes action space in each dimension, and it does so for a specific task (not with the idea of reusability of the learned actions). Although a fixed state action discretization can be reused across tasks, this approach does not provide (1) a compact bank of learned, state-dependent joint motor behaviors, or (2) reusable behavioral components that can be independently frozen or fine-tuned for downstream tasks. In contrast, our reward-free method learns just K=64 state-dependent joint actions, versus 118≈214 million implicit combinations for factorized PPO on Ant-v5.
However, the reviewer’s comment is sensible, and therefore we have followed the recommendation and compared the two methods:
Experimental comparison. Following Tang and Agrawal (2020), we discretized each of the M action dimensions into B=11 equally spaced values and trained factorized categorical PPO using their optimization settings. This represents 11M possible joint actions implicitly, without enumerating them. We used the same downstream task-reward interaction budget and deterministic evaluation protocol as Table 3 (mean return ± 95% CI; 10 seeds).
We find that our frozen token bank achieves comparable mean performance on Ant, higher mean performance on Hopper and Humanoid, and lower performance on Walker2d. Thus, a compact bank of only K=64 learned tokens retains competitive downstream control performance compared to a huge action set comprising BM (118≈214 million on Ant-v5) actions. Therefore, with similar performance, our method provides a very compact set of tokens that can, in addition, can be reused for any downstream task. We will include these results in the revision.