acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Tokenization via Maximum Entropy Mixture Policies with Minimum Entropy Components

Abstract

Standard continuous-control learning in embodied agents entangles movement with reward, producing task-specific policies but no explicit action vocabulary that another controller can directly reuse. We consider the problem of behavioral tokenization: how to learn a fixed, compact bank of state-dependent motor behaviors that becomes a shared discrete interface for future tasks. Previous work has studied this problem using action quantization, sub-policies, options and skills, but no solution provides an unsupervised, task-agnostic method to discover short-lived, action-spanning behavioral tokens. We introduce an online unsupervised framework that discovers behavioral tokens via a joint entropy objective—maximizing the cumulative entropy of an overall mixture policy to ensure rich, exploratory behavior under the maximum occupancy principle, while minimizing the entropy of each component policy to enforce diversity and high specialization. We prove convergence of the value function under our policy-iteration algorithm in the tabular setting, and then extend it to continuous control by fixing the discovered components and deploying them deterministically as quantized actions within an online optimizer to maximize reward. Experiments demonstrate that our maxi-mix-mini-com entropy-based tokenization and action quantization provide reusable behavioral tokens that can be sequenced to achieve competitive performance against continuous-action controllers, largely outperform random action quantization and skill discovery methods, and can be fine-tuned to solve downstream tasks showing superior performance to state-of-art off-policy controllers.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, behavioral tokenization, unsupervised learning, action quantization, continuous control, hierarchical reinforcement learning, entropy maximization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e6ySOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces the maximize mixture, minimize components entropy objective for learning a fixed bank of K state-dependent, one-step motor primitives without task reward, which a downstream discrete controller can then select from. The policy is a state-conditioned Gaussian mixture trained to maximize the cumulative entropy of the mixture while penalizing the entropy of each component ("maxi-mix-mini-com").

For the tabular case, the authors derive alternating closed-form updates for components, weights and responsibilities with monotonic value improvement. For continuous control, the mixture is trained with a SAC-style off-policy algorithm. The component means are then frozen and used as K discrete actions for a discrete PPO controller. Experiments on MuJoCo and other benchmarks compare against continuous PPO, random action sets and unsupervised skill discovery (DIAYN, DADS, LSD, METRA).

Critical strengths

The problem framing is clear and well motivated: a reward-free, reusable, one-step action vocabulary that separates motor learning from task decisions. The evaluation covers many environments and task transfer.

Critical weaknesses
  • Missing discretization baselines. The paper positions itself as online action quantization, and its main claim is that a discrete controller over learned tokens retains continuous-control performance (Table 3). The work cites other methods such as Tang & Agrawal 202, Seyde 2021 or Seo 2025, but does not provide comparisons. The only discretaziation baseline is a random action set.

    Offline discretization methods such as GMM fits on the reward-free explorer are not tested, which would test if learning the action vocabulary online matters.

  • Skill discovery baselines. The results in Tab. 1 are the only quantitative experiments comparing against other skill-discovery methods. From my limited understanding, these methods are optimized for multi step "action chunks", and not suited for single step execution. The ablations in A.7 show, that the proposed method performance halves when holding the same action for 5 steps. Thus, these results could be confounded by the effect of the time scale on the policies.

Questions for authors

How does the frozen token bank compare to an established online discretization baseline, e.g. PPO with a factorized per-dimension categorical policy (Tang & Agrawal, 2020), under the same training budget? If the learned K=16/64 tokens match or outperform this baseline, particularly on Humanoid and Walker2d, this would substantially strengthen the claim that learning the token bank is beneficial.

6 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the efforts and the high quality review of our paper.

Established online discretization baseline under the same budget

We thank the reviewer for suggesting this additional comparison. We would like to clarify first that our objective is to learn a compact, reward-free, state-dependent action vocabulary that can be frozen and reused across downstream tasks while retaining competitive control performance. Factorized categorical PPO (ref: Tang and Agrawal (2020)) addresses a different problem: it factorizes the policy into per-dimension components, and discretizes action space in each dimension, and it does so for a specific task (not with the idea of reusability of the learned actions). Although a fixed state action discretization can be reused across tasks, this approach does not provide (1) a compact bank of learned, state-dependent joint motor behaviors, or (2) reusable behavioral components that can be independently frozen or fine-tuned for downstream tasks. In contrast, our reward-free method learns just K=64K=64 state-dependent joint actions, versus 118≈21411^8 \approx 214 million implicit combinations for factorized PPO on Ant-v5.

However, the reviewer’s comment is sensible, and therefore we have followed the recommendation and compared the two methods:

Experimental comparison. Following Tang and Agrawal (2020), we discretized each of the MM action dimensions into B=11B=11 equally spaced values and trained factorized categorical PPO using their optimization settings. This represents 11M11^M possible joint actions implicitly, without enumerating them. We used the same downstream task-reward interaction budget and deterministic evaluation protocol as Table 3 (mean return ± 95% CI; 10 seeds).

MethodAnt-v5Hopper-v5Walker2d-v5Humanoid-v5
Factorized PPO3432 ± 8871827 ± 3274286 ± 3614499 ± 637
Ours (K=64)3581 ± 323227 ± 1192474 ± 2345672 ± 62

We find that our frozen token bank achieves comparable mean performance on Ant, higher mean performance on Hopper and Humanoid, and lower performance on Walker2d. Thus, a compact bank of only K=64 learned tokens retains competitive downstream control performance compared to a huge action set comprising BMB^M (118≈21411^8 \approx 214 million on Ant-v5) actions. Therefore, with similar performance, our method provides a very compact set of tokens that can, in addition, can be reused for any downstream task. We will include these results in the revision.

AuthorsOpenReview ↗

Response 2/2

Offline GMM discretization baseline

As discussed in Appendix A.1 (lines 723–752), offline fitting begins with a fixed dataset and compresses its existing action support by selecting a low number of representative actions. Once the data are supplied, conditional-GMM fitting is a stationary supervised problem that does not face the exploration problem present in our problem and solution. Indeed, our reward-free online learner must instead generate its own experience while jointly maintaining broad mixture-level action coverage and specialized component identities; consequently, the learned bank and its visitation distribution co-evolve. This is why we treated offline methods as a complementary setting in Related Work.

Following the reviewer’s suggestion again, we ran a matched post-hoc conditional-GMM baseline on Ant-v5. For each of five seeds, our K=1,α=0K = 1,\alpha = 0 maximum-entropy explorer collected 2×1052 \times 10^{5} transitions. Using the same state-conditioned architecture and update count as our learner, we fitted tanh-GMMs with K∈16,64K \in \\{ 16,64\\} offline by conditional negative log-likelihood, retained their deterministic component modes as frozen tokens, and trained the same Table 3 discrete PPO controller for 10710^{7} steps.

BankK=16K=64
Offline conditional GMM (n=5)1505 ± 1951567 ± 238
Ours, Table 3 (n=10)2937 ± 233581 ± 32

Values are mean return ±\pm 95% CI.

Our bank achieves around 2x the GMM return for K=16K = 16 and K=64K = 64, respectively. All GMM runs had plateaued by 5×1065 \times 10^{6} steps, with the best evaluation improving by at most 88 thereafter, and increasing KK produced no clear gain. Conditional likelihood need only fit the overall p(a∣s)p(a\mid s); it does not require every component identity to become distinct and useful after its weights and variances are discarded. Our α>0\alpha > 0 objective explicitly encourages this specialization.

These results indicate that our joint entropy objective yields substantially more useful frozen behavioral tokens than offline conditional maximum-likelihood fitting. In particular, likelihood fitting does not explicitly enforce the component specialization encouraged by our objective.

We will include these results and clarify the distinction between online and offline token learning in the revised manuscript.

Fair temporal horizon for skill-discovery baselines

We agree with the reviewer that DIAYN, DADS, LSD, and METRA are primarily designed to discover temporally extended skills, and this was discussed in our manuscript. Our experiment addresses a different question: Can conventional skill-discovery methods serve as one-step behavioral tokens? To evaluate this, all methods use a discrete PPO controller selecting among 64 frozen tokens or skills at every environment step. Importantly, one-step selection does not prevent temporally extended execution: the controller can, in principle, reproduce a longer skill by repeatedly selecting the same component for 10, 25, or more consecutive steps, but this is rarely done by our learned high level controllers. What our results show is that having one-step composability of tokens adds additional flexibility to better solve downstream tasks that skills do not have.

We will further clarify these crucial distinctions in the revised manuscript.

Reviewer_e6ySOpenReview ↗

Temporal Horizon

I thank the authors for their detailed responses to all reviews. In particular, the additional experiments are highly appreciated and address most of my previous concerns.

I still have some questions regarding Tab. 1 and the comparison against unsupervised skill-discovery methods. I may be misunderstanding some aspects of the baseline methods or experimental setup, so I would appreciate some clarification.

My understanding is that the USD baselines are designed to discover temporally extended skills, which I imagine as sequences of actions (e.g., [a0,a1,a2,...][a_0, a_1, a_2, ...]). I am unsure how these are converted into the 64 frozen choices used by the downstream controller. If selecting the same skill repeatedly simply executes its initial action again, the temporal structure of the original skill would be lost.

Since the USD methods are designed around temporally extended behaviors, I am not convinced that evaluating them as one-step tokens provides a meaningful comparison of their discovery capabilities. I therefore think that the conclusions should be limited to the particular one-step execution setting tested here.

Reviewer_ntveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an unsupervised behavioral tokenization framework that learns a set of short-lived (one-step), state-dependent, action-centric behavioral tokens without task rewards or expert demonstrations. The authors formulate the policy as a mixture policy: the entropy of the mixture distribution is maximized to cover reachable behaviors, while the entropy of each component policy is minimized according to its mixture weight to encourage specialization. For the discrete finite-state/action setting, the paper derives a responsibility-based policy-iteration update; for continuous control, it uses Gaussian component policies, reparameterization gradients, and straight-through Gumbel-Softmax to learn a token bank. After pretraining, the mixture weights are discarded and the component policies are frozen. A new discrete PPO/DQN controller selects tokens step by step and executes their deterministic mean-based actions (called component modes in the paper). The paper also evaluates a setting in which the entire mixture policy is allowed to continue fine-tuning. The experiments include grid worlds, MuJoCo Swimmer/Ant/Humanoid environments, and eight continuous-control benchmarks. With K=64K = 64 frozen tokens, the method outperforms the evaluated unsupervised skill-discovery baselines on Ant and Humanoid and approaches or exceeds continuous PPO on most tasks; the same Ant token bank can be used for motion in different directions and on new maze layouts. The appendix additionally reports random-action quantization, ablations of KK and α\alpha, token duration, a stability-constrained Ant, and SAC fine-tuning experiments. Overall, the problem formulation is clear, the method has some theoretical novelty, and the discrete interface is practical. However, the limitations of the theory, the bias and fairness of the continuous algorithm, and the fact that “reuse” still requires training a new controller for every task call for more cautious interpretation.

Critical strengths
  1. The paper distinguishes short-lived (one-step), state-dependent, action-centric tokens from conventional temporally extended skills/options. Freezing the component policies and letting a higher-level controller select them step by step makes it possible to switch tasks through the same action interface; the mixture-entropy and component-entropy objectives also have a direct information-theoretic interpretation.

  2. The paper provides a high-capacity action-covering result, a responsibility-based iterative update, and a monotonic-improvement proof. The appendix gives derivations of the updates, pseudocode, and a discussion of boundary conditions. The authors also explicitly note that the limiting value can be suboptimal and depend on initialization, making the scope of the theory relatively clear.

  3. The paper evaluates token diversity, random quantization and skill-discovery baselines, KK and α\alpha, downstream learning with frozen tokens, and full-policy fine-tuning. Most tables use 10 random seeds and 95% CIs, and the appendix lists environment versions, training steps, hyperparameters, hardware, and code/container information. Results on new maze layouts and the stability-constrained Ant provide some empirical support for the reuse and fall-resilience claims.

Critical weaknesses
  1. maxi-mix-mini-com entropy combines the maximum entropy of the mixture policy, component-entropy regularization, state-conditioned policies, and action quantization, but these basic ingredients are closely related to existing directions in maximum-entropy RL, mixture policies, skill discovery, and continuous-action discretization. The component policies use a standard tanh-squashed Gaussian parameterization; their deterministic mean-based actions provide the downstream control interface. The joint entropy objective and tabular updates are meaningful contributions beyond this parameterization. A more precise account of their incremental contribution relative to MOP-based exploration and the skill-discovery methods discussed in the paper would strengthen the novelty argument.

  2. Theorem 2 establishes monotonic improvement and convergence of the value sequence under exact tabular updates, without establishing global optimality or convergence of all policy parameters. In continuous control, the method uses Gaussian policies, tanh, finite-sample updates, and a biased straight-through Gumbel-Softmax estimator. The paper acknowledges that the practical off-policy optimizer has no corresponding monotonic-improvement guarantee; the remaining concern is the limited empirical diagnosis of these approximations.

  3. All results are from simulated environments, with the main locomotion benchmarks in MuJoCo and additional Box2D BipedalWalker-v3 and LunarLanderContinuous-v3 benchmarks, as well as MuJoCo FetchReach-v2. Components are executed using deterministic mean-based actions; the performance of sampling components stochastically is not systematically compared. KK must be fixed in advance, α\alpha is selected through a small grid, and each new task requires training a new discrete controller, as the paper explicitly states; reuse is therefore limited to the token bank rather than zero-shot task transfer. The diversity metrics also depend on representative states, MOP trajectories, and uniformly random trajectories, and the paper does not show whether they remain stable under different state distributions, observation noise, dynamics changes, or other embodiments.

Questions for authors
  1. Given Algorithm 2's acknowledged approximations, how sensitive is optimization to replay-state mismatch and the straight-through Gumbel-Softmax estimator?

  2. How does mode-based execution compare with sampling from component policies in terms of action coverage and stability beyond the pretraining state distribution?

  3. Given Table 8 and the acknowledged survival bias, how should KK and α\alpha be selected for tasks requiring goal reaching or deliberate termination?

  4. Including reward-free pretraining and per-task controller training, how many downstream tasks are needed to amortize token-bank learning relative to training continuous policies from scratch?

5 comments
AuthorsOpenReview ↗

Response 1/5

We thank the reviewer for their thorough and thoughtful review.

Contribution and novelty of our work

We apologize for the lack of clarity in our discussion regarding the novelty of our approach. Although the traditional MOP and Max Ent RL approaches indeed contain an entropy term, our objective additionally penalizes the mixture-weighted entropy of the component policies (Eq. 2, proportional to α>0\alpha>0). Without this component-entropy penalty, it would not be possible to address the question of tokenization, which is the central topic of our paper. Therefore, our objective is a step beyond standard exploration. Our objective is also a step beyond standard skill discovery, which aims at discovering temporally extended behavior and not state-dependent distinct behaviors. Our goal is different: we want to discover short-lived tokens that can produce a simple vocabulary that can be flexibly used by a high level controller at any state. We will explain the novelty of our work more clearly in the revised manuscript, also in relation to action quantization.

We have also run tests removing the specialization terms by setting α=0\alpha=0 in the Ant-v5. This ablation shows that without the specialization terms the performance is much lower: the Ant-v5 α=0\alpha=0 ablations produced 1148±3171148\pm317, 1421±1131421\pm113, and 1664±1401664\pm140 for K=16,64,128K=16,64,128 (mean ±\pm 95% CI). Every corresponding α>0\alpha>0 result in Table 8 was higher; the best were 2937±232937\pm23, 3581±323581\pm32, and 3727±333727\pm33. The new row uses the same training and evaluation procedure as Table 8, so the comparison is directly matched. Thus, MOP alone does not produce an equally useful bank. We will include this control in the revision.

Scope of Theorem 2 and continuous-control guarantees

The reviewer is right that our algorithm in the tabular case converges in value function, but it is still possible that the policy parameter continue alternating between locally optimal equivalent policies (for instance, a permutation of the labels, whereby e.g. policy 1 and 2 are swapped and renamed as 2 and 1 respectively). However, these symmetries do not affect the final performance of the algorithm in the tabular case, nor the interpretation of the obtained tokens (as by definition they are defined up to a label permutation).

The reviewer is also right that the continuous implementation does not inherit this monotonic-improvement result. For instance, our Appendix A.3 identifies the straight-through Gumbel–Softmax sample as biased (lines 1424–1432). The discussion preceding Algorithm 2 further states that the continuous theorems are exact on-policy identities, whereas Algorithm 2 uses replay-buffer states, learned critics, target networks, and a straight-through estimator; its updates are therefore off-policy approximations rather than unbiased estimates of the on-policy gradient (lines 2106–2113). Note, however, these quite standard approximations do not preclude our continuous algorithm to perform well in the tested settings and allow for the discovery of behavioral tokens that are useful to solve downstream tasks. We comment more on these approximations in our next response.

We will clarify these points in the corresponding sections.

AuthorsOpenReview ↗

Response 2/5

Replay-state mismatch and straight-through Gumbel–Softmax sensitivity

Thanks for the question. In this response, we examine these optimization choices more closely with extended analysis and provide additional insight into the behavior of our method and its implementation. We conducted a newpaired five-seed sensitivity study on Ant-v5 with K=64K=64, α=0.2\alpha=0.2, and 2×1052\times10^5 pretraining interactions, varying one factor at a time around the submitted configuration. We restrict this empirical conclusion to Ant-v5, K=64K=64, α=0.2\alpha=0.2, and the tested settings.

Replay-state mismatch. We compared FIFO replay capacities of 10410^4, 10510^5 (the setting used in the previously submitted version; Table 9, lines 2025–2041), and a full-history buffer containing all 2×1052\times10^5 interactions. The full-history setting produced similarly stable training and final-bank diagnostics. In the detailed 10410^4-versus-10510^5 comparison, reducing the capacity from 10510^5 to 10410^4 reduced mean replay age from 5×1045\times10^4 to 5×1035\times10^3 interactions, while KeffK_{\mathrm{eff}} remained similar (56.856.8 versus 54.754.7) and MPED\mathrm{MPED} was nearly unchanged (0.3770.377 versus 0.3800.380). The replay-minus-fresh gaps were also negligible: for 10410^4 and 10510^5, respectively, ΔR^α\Delta\widehat R_\alpha was 0.0170.017 and 0.0380.038, while ΔI^(A;C∣S)\Delta\widehat I(A;C\mid S) was −0.004-0.004 and approximately 0.0000.000. The only detected difference in that comparison was a modest 0.1330.133-nat reduction in mutual information with the 10410^4 buffer. Thus, replay capacity had little empirical effect across the tested range from recent replay to full-history replay, although Algorithm 2 remains an off-policy approximation as explicitly stated in lines 2106–2113.

Straight-through temperature. The moderate values τg=1\tau_g=1 and τg=2\tau_g=2 produced closely matching objective, mutual-information, component-utilization, and separation results. At the submitted τg=2\tau_g=2, all 6464 components remained sampled and Keff=56.8K_{\mathrm{eff}}=56.8. The lower value τg=0.5\tau_g=0.5 remained numerically stable but learned a lower-utilization bank (Keff=22.3K_{\mathrm{eff}}=22.3), whereas τg=5\tau_g=5 destabilized component utilization and two of five seeds collapsed to one effective component.

Importantly, for fixed logits and Gumbel noise, τg\tau_g does not alter the hard component selected in the forward pass; it controls only the soft relaxation used during backpropagation (Appendix A.3, lines 1424–1432). Therefore, the degradation at τg=5\tau_g=5 reflects a poor setting for this straight-through optimization estimator rather than a failure of the maxi-mix-mini-com objective. The agreement between τg=1\tau_g=1 and τg=2\tau_g=2 shows that the submitted value lies within a stable range rather than being an isolated tuned choice.

We will report the previously omitted fixed temperature τg=2\tau_g=2, clarify that it is not annealed, distinguish it from the target-network smoothing coefficient, and include these sensitivity results in the revision.

AuthorsOpenReview ↗

Response 3/5

Deterministic mode execution versus stochastic component sampling

Thanks again for the question. In our paper we have chosen to use the mode of the token policies and remove their sampling noise, rather than using them with the stochastic component. Our motivation was that this choice would make it simpler for a high level controller to combine them to solve downstream tasks. We next tested and confirmed this intuition:

We froze an Ant-v5 bank trained with K=16K=16 and α=0.1\alpha=0.1, then trained a separate PPO controller with identical settings for each c∈0,0.1,0.25,0.5,1.0c\in\\{0,0.1,0.25,0.5,1.0\\}, which controls how much noise is taken out from the token. The selected token executes at=tanh⁡(μkt(st)+c,σkt(st)ϵt)a_t=\tanh(\mu_{k_t}(s_t)+c\\,\sigma_{k_t}(s_t)\epsilon_t), with the same cc during training and evaluation. Thus, c=0c=0 is deterministic mode execution and c=1c=1 restores fully stochastic component sampling. Final forward-locomotion returns are mean ±\pm 95% CI:

c00.10.250.51.0
Final return1669.4 ± 22.221318.0 ± 379.6554.9 ± 272.2177.1 ± 20.790.7 ± 13.0

We found that, as predicted, performance decreased as component stochasticity was restored. Because each cc used a separately trained controller, this is not an evaluation-only noise effect; it is a comparison under the task-induced downstream state distribution. Stochastically running a token increases local within-token support, but the bank already provides coverage through distinct state-conditioned modes (Sec. 4, lines 369–373; Table 6, lines 1890–1895), and the additional spread did not improve downstream control.

A separate survival test that we have run next supports the same conclusion: every fixed deterministic mode reached 1000±01000\pm0 steps, whereas sampling from a fixed (but stochastic) component achieved 787±104787\pm104; Videos 2 and 3 provide representative rollouts.

We will include the controlled execution comparison in the revision and restrict this conclusion to the tested Ant setting.

Selecting K and α

We agree that the current method does not automatically select KK or α\alpha. KK determines the capacity of the token interface, while α\alpha controls component specialization. Nevertheless, Table 8 provides useful practical guidance. At the fixed α=0.2\alpha = 0.2, we obtain the following Ant-v5 returns (mean ±95\pm95\\% CI):

MethodAnt-v5 return
Continuous PPO3176 ± 224
Ours (K=16)2937 ± 23
Ours (K=64)3581 ± 32
Ours (K=128)3399 ± 20

Thus, the same α\alpha provides competitive control across substantially different bank sizes, showing that selecting a different value for every KK is not necessary to obtain an effective token interface. Together with Table 3, where K=64K = 64 matches or exceeds continuous PPO on seven of eight benchmarks, these results support K=64K = 64 as a reliable practical default. The K=64K = 64 bank size also performed well in the full-mixture fine-tuning experiments across Ant, Hopper, and Walker2d (Table 2).

Reviewer_8LCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an unsupervised method for learning reusable, state-dependent behavioral tokens by maximizing mixture-policy entropy while penalizing component entropy. The components are then frozen and selected by a discrete downstream controller. The paper provides a tabular convergence result and evaluates token reuse and policy fine-tuning across continuous-control tasks.

Critical strengths
  1. The frozen, one-step token interface is useful and well motivated. Reusing the same Ant bank across different reward objectives and a new maze layout provides meaningful evidence of transfer.

  2. The paper combines a monotonic value-improvement result with experiments across eight control environments. The strong Humanoid results suggest that a compact token bank can retain substantial control capability.

Critical weaknesses
  1. The continuous-control experiments lack an α = 0 ablation using the same mixture architecture and training procedure. Without this comparison, the benefit of the proposed component-entropy penalty remains unclear. The random-initialization baseline establishes the value of pretraining, but does not isolate the proposed objective.

  2. The efficiency comparisons give the proposed method additional interactions. For example, Table 2 compares 300,000 pretraining plus 100,000 downstream steps against 100,000 baseline steps. This supports faster learning with task rewards, but not better overall sample efficiency. Comparisons using equal total interaction budgets are needed to establish the broader efficiency claim.

Questions for authors
  1. How does downstream performance change when α = 0, with the architecture and pretraining budget held fixed?

  2. Do the gains remain with equal total interaction budgets and matched initial exploration schedules?

  3. The conclusion mentions variance downscaling. What factor is used, where is it applied, and how sensitive are the results to it?

  4. Since downstream control discards the pretrained mixture weights, what evidence supports the safety of individual tokens rather than only the pretrained mixture?

5 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for their reading and constructive questions.

α = 0 ablation

We apologize for not clearly explaining the role of α\alpha in our objective. Setting α=0\alpha=0 will predictably generate components that are not distinct from each other. In other words, setting α>0\alpha>0 is critical for the specialization of the components. More specifically, in Appendix A.2.5 (Eq. 45, lines 1213–1223) we rewrite our objective as

R(s;π,w)=(1−α)H(A∣S=s)+αI(A;C∣S=s).R(s;\pi,w)=(1-\alpha)H(A\mid S=s)+\alpha I(A;C\mid S=s).

Here, C is the component index and A the resulting action. At α = 0, this objective reduces to maximum-entropy MOP and depends only on the marginal mixture policy; it is therefore indifferent to its component decomposition, so a fully collapsed bank can be optimal. For α > 0, the mutual-information term favors components that produce distinguishable actions at the same state. Thus, α directly controls the trade-off between broad action coverage and distinguishable token identities. We will further clarify this in the revision.

To complement the above response, we have followed the reviewer’s recommendation and tested the ablation with α=0\alpha = 0. We ran the α = 0 ablation for all three bank sizes K=16, 64 and 128, using the corresponding architecture and the same training and evaluation procedure at each K, with only α changed. The returns were 1148 ± 317 for K = 16, 1421 ± 113 for K = 64, and 1664 ± 140 for K = 128 (mean ± 95% CI). At every tested KK, all positive-α\alpha settings in Table 8 outperformed the best result obtained with α=0\alpha = 0: the lowest α > 0 returns were 1669 ± 22, 2399 ± 19, and 3399 ± 20 for K = 16, K = 64, and K = 128, respectively (lines 1963–1972). Across all three tested bank sizes in Ant-v5, these results provide direct evidence that the component-specialization term improves downstream performance beyond maximum-entropy pretraining alone. We will include the complete ablation in the revised manuscript.

Ant-v5 downstream return (mean ±\pm 95% CI). The α=0\alpha = 0 row reports evaluation returns from the new ablations. The α>0\alpha > 0 rows reproduce the submitted Table 8 results.

αK=16K=64K=128
01148 ± 3171421 ± 1131664 ± 140
0.11669 ± 222399 ± 193727 ± 33
0.22937 ± 233581 ± 323399 ± 20
0.31864 ± 183281 ± 233539 ± 31
AuthorsOpenReview ↗

Response 2/3

Equal total interaction budgets and matched exploration

The reviewer is correct that pretraining seems to have an advantage over SAC in our comparisons in terms of the allowed sample budget. This advantage, however, vanishes when using an increasing number of downstream tasks, because the pretraining budget is fixed and does not grow with the number of downstream tasks tested. Our rationale was that a fair comparison happens when the same number of reward-interactions are given to any method, not when the total pretraining + reward-interaction budget are equated (i.e., comparing SAC with way more reward iterations against our method would of course give a huge advantage to the former). Indeed, our approach followed standard procedures (Laskin et al., 2021), which reports pretraining separately and compares methods under the same downstream task-reward budget.

To better compare all the methods on an equal footing, we propose that we must use approximately the same number of interactions with the task(s) and approximately the same number of pretraining + task interactions for each method tested. This can be accomplished by having a fixed pretraining phase, and then testing over multiple downstream tasks. Therefore, we have now compared the performance of our method against SAC when we fix the amount of training samples, but when the number of downstream tasks to be solved is 4 and 8 (different motion directions: the four cardinal directions and eight directions spaced at 45∘45^\circ intervals).

For each seed, we pretrained one Ant mixture once for 300K300\mathrm{K} reward-free interactions, copied this initialization across the downstream tasks, and fine-tuned the complete mixture separately for 100K100\mathrm{K} rewarded interactions per task. SAC was trained independently for 150K150\mathrm{K} rewarded interactions per task using the same initial exploration schedule.

Task suiteOursSAC
4 cardinal directions1013 ± 238755 ± 208
All 8 directions990 ± 219823 ± 185

Values are mean final return ±95\pm95\\% CI over ten seed-level task-suite averages. Total interactions are equal at six tasks: 300K+6(100K)=6(150K)=900K300\mathrm{K}+6(100\mathrm{K})=6(150\mathrm{K})=900\mathrm{K}. Thus, the four- and eight-task settings bracket the amortization point.

Our method achieves a higher mean return in both settings. Notably, across eight tasks, it uses 1.1M1.1\mathrm{M} total interactions compared with 1.2M1.2\mathrm{M} for SAC, while SAC receives 5050\\% more task-reward interactions per task. We will include these results in the revision.

Laskin, M. et al. (2021). URLB: Unsupervised Reinforcement Learning Benchmark. NeurIPS 2021 Datasets and Benchmarks Track. https://arxiv.org/abs/2110.15191

AuthorsOpenReview ↗

Response 3/3

Variance downscaling factor and sensitivity

Again, we are sorry for the lack of clarity in our explanation. By “variance downscaling,” we refer to replacing each component variance σk2(s)\sigma_k^2(s) by c2σk2(s)c^2\sigma_k^2(s) during execution, with 0≤c2≤10\leq c^2\leq 1, following the terminology used for GMM policies (Sheebaelhamd et al., 2025). During reward-free pretraining, we sample from the full learned component distribution without scaling its standard deviation: zt=μk(st)+σk(st)ϵtz_t=\mu_k(s_t)+\sigma_k(s_t)\epsilon_t, at=tanh⁡(zt)a_t=\tanh(z_t), where ϵt∼N(0,I)\epsilon_t\sim\mathcal N(0,I) (Appendix A.6.1, lines 1552–1560). In all submitted downstream experiments, the selected token is instead executed deterministically as at=tanh⁡(μkt(st))a_t=\tanh(\mu_{k_t}(s_t)) (Sec. 4, lines 369–373; Appendix A.6.1, lines 1569–1571). This corresponds to a standard-deviation multiplier c=0c = 0; pretraining uses c=1c = 1. We will clarify this in the new version.

To measure sensitivity to stochastic token execution, we have run now more analysis by freezing reward-free K=16,α=0.1K=16,\alpha=0.1 Ant-v5 banks and training a separate downstream PPO controller for each c∈0,0.1,0.25,0.5,1.0c\in\\{0,0.1,0.25,0.5,1.0\\}, executing at=tanh⁡(μkt(st)+cσkt(st)ϵt)a_t=\tanh(\mu_{k_t}(s_t)+c\sigma_{k_t}(s_t)\epsilon_t). The same multiplier was used during training and evaluation. Here, cc scales the standard deviation; the corresponding covariance multiplier is c2c^2. Final returns are mean ±95\pm 95\\% CI:

c00.10.250.51.0
Final return1669.4 ± 22.221318.0 ± 379.6554.9 ± 272.2177.1 ± 20.790.7 ± 13.0

We find that mean performance decreases sharply as stochasticity is restored, particularly for c≥0.25c\geq0.25. Thus, this experiment shows that downstream performance in this Ant-v5 setting is sensitive to stochastic component sampling and supports the deterministic mode execution used in the paper. We will replace the ambiguous phrase “variance downscaling” with “deterministic mode execution” and include this sensitivity result in the revision.

Sheebaelhamd, Z., Tschannen, M., Muehlebach, M., and Vernade, C. (2025). Quantization-Free Autoregressive Action Transformer. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).

Safety of individual tokens after discarding mixture weights

Thanks for the question. Our experience was that all tokens were remarkably stable when run deterministically (see previous point) and fixed, without switching between them allowed (see Fig. 3 and videos 2). When run stochastically, we also found that the tokens were remarkably stable and provided safe behaviors, without falls (see Videos 3).

To quantify these observations, we have now run a survival test on each of trained K = 16 bank tokens. We find that the deterministic pretrained mixture (deterministic execution, c=0, of both mixture weight and component execution) achieves 100% survival, with no single episode or token producing a fall (mean episode length = 1000 ± 0, where the maximum horizon of the episodes was 1000 steps and we run 1000 episodes per token; error is 95% CI ). More importantly, after discarding the mixture weights and holding each individual token mode fixed for the entire episode, every one of the 16 tokens also achieves a mean life duration of 1,000 ± 0. Randomly switching among token modes at every step, again without using the learned mixture weights, likewise achieves 1,000 ± 0.

For completeness, we also tested the stability of the token when run stochastically (c=1), not with their mean/mode (c=0). For the fixed-component condition, we evaluated each of the 16 components over 100 episodes and computed the mean and 95% CI across the component-level means (1,600 episodes total). As expected, stochastic execution produced shorter survival: the stochastic pretrained mixture achieved 815 ± 92, while the evaluation that held each component fixed and sampled its actions achieved 787 ± 104. Therefore, the tokens are remarkably safe even when run forward fully stochastically.

We will include these relevant results in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.