acceptodds
Under review as a conference paper at ICLR 2027

Efficient Implicit Boltzmann Policy Optimization via Mixture Likelihood Maximization

Abstract

Online reinforcement learning often requires representing complex, potentially multimodal action distributions induced by the value function. Recent diffusion and flow-based policies have substantially improved policy expressivity by enabling flexible approximations to complex action distributions. However, it remains challenging to achieve both high sampling fidelity and low inference latency with expressive generative policies, as they typically generate actions iteratively. While recent one-step approaches reduce inference cost, training these expressive policies has remained computationally expensive. We introduce **Efficient Implicit Boltzmann Policy Optimization via Mixture Likelihood Maximization (iBOLT)**, which efficiently trains a one-step policy to approximate the value-induced Boltzmann distribution. iBOLT represents the policy as a continuous mixture of truncated Gaussians, with component parameters generated from the state and a latent variable by a single network. Then, we use a tractable finite-mixture approximation to reweight sampled actions toward the Boltzmann target and train the policy by maximizing the weighted mixture likelihood. During deployment, the learned policy generates actions with a single network evaluation, without iterative refinement. Experiments on MuJoCo benchmarks and multimodal tasks demonstrate that iBOLT achieves performance competitive with state-of-the-art generative policy methods, while substantially reducing both training cost and inference latency.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Implicit Policy, Boltzmann Policy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZHgiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents iBOLT (not a 1:1 acronym for what it actually does), an approach that builds on recent advances in 1-step generative policies, and frames it in two interesting ways. First, it creates a Boltzmann target that more naturally captures multi-modal action distributions (or state-action value distributions). Second, it creates a set of loss functions based on importance sampling (and the aforementioned Boltzmann style target) that re-cast the problem as an implicit policy optimization one. The paper -- cleanly, in my opinion -- describes this narrative both mathematically and algorithmically, and then proceeds to an empirical evaluation on robotics or navigation style tasks that are either (a) know to be multi-modal and/or (b) are standard in the RL literature.

Critical strengths
  • The paper is organized well and the prose is generally concise and clear.

  • The problem is well-motivated in terms of both established and emerging methods in ML broadly and RL specifically. For example, it melds advances in multi-model representations (and doing it in 1 step) which have gained popularity relatively recently, along with now-class approaches that use implicit parameterizations from the RL literature.

  • The story in section 4 is nice. It proceeds in a clear way from well-known "tricks" and methods in order to derive the critic loss function and ultimately the policy extraction method in (15). There derivation is sound, and the subsequent discussion (at the end of section 4) brings things to a close in a concise, conceptual way.

  • The questions that kick off section 5 do a nice job of clearly identifying and driving the agenda of the paper. These questions are also essentially the contributions; as such they are mentioned in section 1. However, I suggest the authors consider simply adding these questions to the intro; this is not a requirement, just a suggestion that would perhaps only take a bit of light editing.

Critical weaknesses
  • The evaluations are largely on "toy" problems. These are, however, common in RL literature. The issue is that claims about multi-modality, expressitivity, etc are limited (at least empirically).

  • The experiments do well to try to evaluate the paper's central claims. However, I believe the paper is not rigorous enough (or, perhaps, descriptive enough) regarding the design choices of the architectures. This includes choices for iBOLT itself but more importantly surrounds the various baselines and their implementation.

Questions for authors
  1. The TD loss in equation (4) is basically standard, but for clarity it would help to be explicit about how to get from the two sets of parameters learned in (4) to the Boltzmann target in (5), which only contains one Q network. It is in the pseudocode in the appendix, perhaps say how this target update works in one conceptual sentence.

  2. In section 5, the baseline and iBolt architectures use the same hyperparameters, especially when the objective of an experiment is to isolate things that are outside of the architecture. However, some of the hyperparameters seem to be selected arbitrarily (e.g. 128 latent samples and candidate actions). Is this number optimized? Would a different number, say, benefit reverse-KL training? It is difficult to know without parameter sweeps.

  3. Similarly, for the GMM40 multimodal recovery study, it is unclear how each method has been optimized (or not) in terms of hyperparameters? Do the baselines use off-the-shelf designs? Has iBOLT been tuned? For some of the Mujoco tasks, baseline architectural details are shown in the appendix. However, these are all the "same" or similar. I assume this is to isolate for compute and attempt to get 1:1 comparisons in terms of computational resources. However, these hyperparameters might happen to be optimal for iBOLT and severely suboptimal for the baselines.

  4. Why isn't MEoW included in the first study? Or more generally, why aren't the baselines the same for every experiment? It does not seem like there is anything fundamentally constraining, either in terms of the RL algorithms or the environments, to prevent this.

  5. What are the error bars -- or σ\sigma -- for Table 2?

4 comments

Response to Reviewer ZHgi

We greatly appreciate the reviewer's thoughtful and constructive feedback. We are especially grateful for the helpful suggestion on the Introduction: we have incorporated the suggested questions into the revised Introduction. Our responses to the remaining points are given below.

[W1: Evaluation beyond toy problems]

In the revision, we provide additional evidence that the learned policies are multimodal on standard continuous control benchmarks. On Hopper and Humanoid, we measure the number of persistent modes of the learned action distributions in the full action space, over 100 randomly selected visited states. iBOLT has two to three modes on average, while the tanh-Gaussian policy trained with the same critic has a single mode (Appendix E.3, Tables 8 and 9, Figure 16).

[W2: Design choices for iBOLT and the baselines]

For iBOLT, the main components follow from the derivation in Section 4 and are not separate design choices. The semi-implicit actor is introduced for expressivity, the finite-mixture approximation handles its intractable marginal density and gives an upper bound on the forward KL (Proposition 1), and self-normalized importance sampling gives a consistent estimate of the expectation under the intractable Boltzmann target. The remaining implementation choices are listed in Appendix C.4 (Table 4) and ablated in Appendix D: the temperature (D.1), the exploration noise (D.2), and the numbers of latent and reference samples (D.3).

For the baselines, we selected a representative method from each family of expressive policies for online RL: diffusion (DIPO), flow matching (MFPO), normalizing flow (MEow), and implicit sampling (SQL), together with the standard SAC and TD3. Each is run from its official implementation (Appendix C.1), and Appendix C.6 (Table 5) lists their architectures and hyperparameters, where all MLP-based methods share a 2×2562\times256 MLP backbone to match network capacity. In the revision, we also added the tuning procedure for iBOLT and the baselines, the evaluation rule of each baseline, and the timing protocol (Appendix C.6).

[Q1: From the two critics in Eq. (4) to the Boltzmann target in Eq. (5)]

Thank you for pointing this out. We train two critics Qϕ1Q_{\phi_1} and Qϕ2Q_{\phi_2} with the loss in Eq. (4), where the minimum over the two target critics gives the clipped double-Q target. For policy learning, the Boltzmann target in Eq. (5) uses the mean of the two online critics, Qˉϕ(s,a)=12(Qϕ1(s,a)+Qϕ2(s,a))\bar Q_\phi(s,a) = \frac{1}{2}\big(Q_{\phi_1}(s,a) + Q_{\phi_2}(s,a)\big). The target critics are updated as an exponential moving average of the online critics (line 15 of Algorithm 1). We added this explanation to Section 4.2 and now write Eq. (5) with Qˉϕ\bar Q_\phi.

[Q2: Sweep over NN and MM for the forward- and reverse-KL experiment of Figure 2]

The value N=M=128N=M=128 is a default and was not optimized. To see its effect, we ran the full grid N,M∈{32,128,512}N, M\in\{32,128,512\} for both objectives on the two targets of Figure 2: the three-mode Gaussian mixture (Figure 2a-b) and the spike-and-ramp target (Figure 2c-d), with three seeds per setting. The table reports the mean total variation (TV) distance to the target and, in brackets, the number of seeds that miss at least one mode.

NNMM3-Gaussians: FKL3-Gaussians: RKLSpike+ramp: FKLSpike+ramp: RKL
32320.6667 [3/3]0.6666 [3/3]0.0618 [0/3]0.2027 [3/3]
321280.0528 [0/3]0.6666 [3/3]0.0644 [0/3]0.2037 [3/3]
325120.0459 [0/3]0.6666 [3/3]0.0894 [0/3]0.1997 [3/3]
128320.2535 [1/3]0.6666 [3/3]0.0642 [0/3]0.2040 [3/3]
1281280.0324 [0/3]0.6666 [3/3]0.0406 [0/3]0.1761 [2/3]
1285120.0433 [0/3]0.6666 [3/3]0.0626 [0/3]0.1996 [3/3]
512320.0548 [0/3]0.6666 [3/3]0.0448 [0/3]0.2054 [3/3]
5121280.0431 [0/3]0.6666 [3/3]0.0544 [0/3]0.1444 [2/3]
5125120.0253 [0/3]0.6666 [3/3]0.0267 [0/3]0.1513 [1/3]

Our forward-KL training recovers all modes in every seed once M≥128M\ge128. In contrast, reverse-KL training misses a mode in 50 of 54 runs, including every run on the three-mode Gaussian mixture target. The conclusion of Figure 2 therefore does not depend on the specific choice of 128. For reverse-KL training, NN and MM set the number of sampled latents and actions per group, and the marginal density is estimated separately with L=217L=2^{17} latent samples.

[Edit: Updated figure, table, and appendix references to match the revised manuscript. The reported results are unchanged.]

[Q3: Hyperparameters and fairness of the comparisons]

For GMM40, each baseline keeps the architecture and sampling procedure of its official implementation, and all methods share the target (Q=log⁡pGMM40Q=\log p_{\mathrm{GMM40}} with temperature 1), optimizer, learning rate, batch size, and number of updates (Table 3). iBOLT was tuned on this task: N=M=256N=M=256 was chosen from the sweep in Appendix D.3 (Table 7), and the scale bounds and initialization in Table 3 are specific to GMM40. The baselines use the same hyperparameters as in the MuJoCo experiments, since tuning them on GMM40 did not change the results significantly.

For MuJoCo, the shared settings in Table 5 are the standard settings of SAC and TD3, which iBOLT adopts. They are therefore not values tuned for iBOLT. For iBOLT, only the temperature τ\tau is tuned, from six values with two seeds, and N=M=64N=M=64 is fixed across tasks without a search. SAC, TD3, and DIPO use their published hyperparameters, and MEow keeps its own flow-based architecture and official hyperparameters. For MFPO and SQL, whose official settings use a different backbone, we additionally searched the target-entropy coefficient over (1/4, 1/2, 1, 2, 4)×\times and the reward scale over (1/3, 2/3, 1, 2, 3)×\times the default of the official implementation. The default was best in both cases.

[Q4: Baseline selection across experiments]

The MuJoCo comparison includes all six baselines (SAC, TD3, MEow, SQL, MFPO, DIPO). The other two experiments each omitted some of them, and we have now added the missing ones.

In GMM40, TD3 is not applicable because its policy is deterministic and cannot represent a target distribution. MEow was not included in the original study. We have now added it under the same setting as the other baselines, using its released architecture and objective (Figure 3):

MethodMMD ↓Mode coverage ↑, out of 40
iBOLT0.0260 ± 0.005040.0 ± 0.0
MEow0.3029 ± 0.101924.0 ± 10.15

In PointMaze, DIPO was not included in the Figure 5. We now report it on both the Medium and Hard mazes:

MethodMedium success (%)Medium goals / 4Hard success (%)Hard goals / 8
iBOLT100.0 ± 0.04.0 ± 0.0100.0 ± 0.08.0 ± 0.0
SAC100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0
TD360.0 ± 54.80.6 ± 0.580.0 ± 44.70.8 ± 0.4
MEow100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0
SQL5.1 ± 5.91.6 ± 0.920.2 ± 13.64.8 ± 2.2
MFPO91.5 ± 15.41.0 ± 0.035.6 ± 48.90.4 ± 0.5
DIPO100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0

DIPO reaches 100% success on both mazes with a single goal per seed, as SAC and MEow do, while iBOLT reaches all four goals in Medium and all eight goals in Hard.

[Q5 : Error bars in Table 2]

We added them in the revision. Table 2 now reports the latency as mean ± standard deviation over 5 seeds for every method (for example, 0.20±0.010.20\pm0.01 ms for iBOLT). For each seed, we first generate 300 actions as warm-up and then measure the latency of the next 3,000 actions. The full measurement protocol is given in Appendix C.6.

I thank the authors for their thorough rebuttal. The new results in Appendix E are indeed quite interesting.

I hope the authors feel that the paper is now a bit more clear and precise, and overall "better". It seems to me that it is.

I stand by my position that current standard benchmarks are probably insufficient for empirical arguments regarding claims of complexity, multi-modality, etc. This is not a criticism of the authors or the paper; it is simply a statement about the state of things.

I am not asking for another revision; however, I would like to note that many of the claims are about multi-modality, and the main body text has been improved to reflect this. I am not asking the authors to include (for example) App E.3 in the main body, but I suggest in the final camera-ready version the main body at least directs the reader to this and other details (as it already does for most other parts of the appendix).

Dear Reviewer ZHgi,

Thank you again for your thoughtful follow-up and encouraging comments. Your constructive feedback throughout the review process has helped us improve the clarity and precision of our paper.

Following your suggestions, we clarified the construction of the Boltzmann target, expanded the sample-count ablations and baseline comparisons, and provided more detailed experimental settings and latency statistics. Together with the additional multimodality analyses, these revisions have helped us present a clearer and more comprehensive evaluation of iBOLT. We also appreciate your observation that current standard benchmarks have limitations in fully assessing multimodal policy representations, and agree that this could be an important direction for future research.

Regarding your suggestion on the presentation of these results, should we have the opportunity to prepare a final camera ready version of the manuscript, we will further revise the main text to explicitly direct readers to the multimodality analysis in Appendix E.3 and the other newly added results in Appendix E.

Thank you once again for your time and thoughtful review.

Best regards, Authors

Reviewer_uAXnOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes iBOLT, a one-step semi-implicit policy represented as a continuous mixture of truncated Gaussians. It approximates the marginal density using a finite mixture and fits a critic-induced Boltzmann distribution through self-normalized importance-weighted likelihood maximization. The analysis establishes a finite-mixture forward-KL upper bound and characterizes its leading-order regularization effect. Experiments investigate multimodal distribution fitting, behavioral diversity, and the return–computation trade-off on five MuJoCo tasks.

Critical strengths

1.The method offers a simple approach to expressive policy learning without iterative generation or specialized flow architectures. The finite-mixture analysis clarifies the relationship between the marginal-policy objective and its tractable surrogate, including how the number of latent samples affects component specialization. 2.The empirical evaluation addresses both distribution fitting and control performance. The same-architecture KL comparison, reported recovery of all 40 GMM components, and five-seed MuJoCo results are useful evidence. Temperature and exploration-noise ablations further strengthen the evaluation, while the comparison with JAX-based MFPO supports the practical efficiency argument.

Critical weaknesses
  1. The GMM40 proposal specification is unclear. Section 4.3 and Algorithm 1 sample from the actor’s finite mixture. However, Table 3 introduces a proposal standard-deviation floor of 0.050.05, exceeding the maximum actor standard deviation, e−3.5≈0.0302e^{-3.5} \approx 0.0302. If applied as written, this produces a different proposal. The paper does not explain how this change enters the sampling and density correction, leaving uncertainty about whether the experiment implements the stated objective.
  2. The main control evaluation uses a different policy from the stochastic actor being studied. Appendix C.6 evaluates a=tanh⁡(μθ(s,0))a = \tanh(\mu_\theta(s, 0)), fixing the latent and removing conditional noise. This is generally neither the mixture mean nor its mode. The reported returns therefore characterize a deterministic slice of the generator; sampled-policy results are needed to connect these findings to the claimed expressive Boltzmann policy.
  3. Online comparisons do not isolate the improvement over prior semi-implicit policies. IDAC is discussed but not evaluated, and the same-architecture forward-/reverse-KL comparison is restricted to fixed one-dimensional targets. A matched semi-implicit baseline on the control tasks would better establish the contribution beyond the existing policy representation.
Questions for authors
  1. How is the GMM40 proposal floor applied, and which density enters the importance-weight denominator? Does removing the floor preserve the reported mode coverage?
  2. What returns and inference latency are obtained when sampling both the latent and conditional Gaussian? What motivates selecting z=0z = 0 for deterministic evaluation?
  3. How does iBOLT compare with IDAC or a matched reverse-KL semi-implicit actor on the control benchmarks under comparable training budgets?

4. You are missing some important baselines, particularly CrossQ. This is especially important because you emphasize the multimodal capabilities introduced by the GMM mixture. However, can your method actually outperform a much simpler baseline like CrossQ?

6 comments

Response to Reviewer uAXn

We sincerely thank the reviewer for the valuable feedback. The comments have helped us improve both the clarity of the method and the completeness of the experiments, and we respond to each point below.

[W1, Q1 : GMM40 proposal specification]

Thank you for thoughtful feedback. The “proposal standard-deviation floor” row in Table 3 was left over from an earlier configuration and was not used for the reported results. We removed it in the revision.

In the reported GMM40 runs, the sampling density qq in Eq. (10), the density qq in the denominator of the importance weights in Eq. (11), and the density π^θ,N\hat \pi_{\theta, N} whose log-likelihood is maximized in Eq. (14) are all the same finite mixture defined in Eq. (7). Thus, the reported experiment implements the objective of Section 4.3 and Algorithm 1 as stated, with no floor applied to the proposal.

For reference, we also ran GMM40 with the proposal qq broadened with a standard-deviation floor of 0.05. This broadened mixture is used for both sampling and the importance-weight denominator, and π^θ,N\hat \pi_{\theta, N} (without the floor) is used only in Eq. (14). Even though the proposal is no longer the same as the fitted policy mixture, the learned policy still covers all 40 modes in every seed, with a similar MMD.

Proposal standard-deviation floorMMD ↓Mode Coverage ↑
Disabled (paper)0.0260 ± 0.005040/40
0.050.0276 ± 0.005740/40

[W2, Q2: Evaluation with sampled actions and the choice of z=0z=0]

Following the common practice of deterministic evaluation on MuJoCo benchmarks, we deterministically evaluate the MuJoCo returns with z=0z=0, the mean and mode of the latent prior. Several baselines (SAC, TD3, and MEow) are also evaluated deterministically in our comparison, as in their official implementations. The z=0z=0 rule is used only for evaluating the MuJoCo returns. During training, data is collected with actions sampled from the full mixture, as in Algorithm 1. In the multimodality experiments (GMM40 and PointMaze), actions are sampled stochastically with z∼N(0,I)z\sim\mathcal{N}(0,I) for iBOLT and from the stochastic policies of the baselines.

As the reviewer requested, we added the MuJoCo returns obtained when actions are sampled from the policy in Appendix E.1 (Figure 12, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/MuJoCo_zEval.png). We evaluate the same trained policies under three rules: (i) z=0z=0, which executes tanh⁡(μθ(s,0))\tanh(\mu_\theta(s,0)) and is used in all other MuJoCo results, (ii) z∼N(0,I)z\sim\mathcal{N}(0,I) sampled at every step, executing the component center tanh⁡(μθ(s,z))\tanh(\mu_\theta(s,z)), and (iii) the full mixture, where both the latent and the conditional Gaussian kθ(⋅∣s,z)k_\theta(\cdot\mid s,z) in Eq. (2) are sampled.

Sampling zz alone gives nearly the same returns as z=0z=0, so the reported returns are not specific to the z=0z=0 slice of the generator. Sampling the full mixture lowers the returns on some tasks. This is the usual gap between stochastic and deterministic evaluation, so we keep z=0z=0 as the default evaluation rule.

All three rules use a single network evaluation, and sampling the full mixture increases the latency only from 0.204 ms to 0.225 ms.

[W3, Q3: Comparison with IDAC and a matched reverse-KL semi-implicit actor]

Thank you for this suggestion. We added a comparison with IDAC [Yue et al., 2020] in Appendix E.2 (Figure 13, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/MuJoCo_IDAC.png). IDAC differs from iBOLT in two components: its actor is trained with a reverse-KL objective, and its critic is an implicit distributional critic. To separate the two, we compare three variants under the common protocol of our MuJoCo comparison, keeping IDAC’s own reported hyperparameters: (i) IDAC, the full method, (ii) SIA, IDAC’s semi-implicit actor and reverse-KL objective trained with the same twin scalar critic as iBOLT, which is the matched reverse-KL baseline, and (iii) iBOLT + IDAC critic, iBOLT’s actor and objective trained with IDAC’s distributional critic. All three are ported from the official IDAC implementation.

The four methods form a two-by-two comparison of the actor (IDAC’s reverse-KL actor or iBOLT’s forward-KL actor) and the critic (twin scalar or IDAC’s distributional critic). With the scalar critic, iBOLT achieves higher returns than SIA on Hopper, Walker2d, HalfCheetah, and Ant, and learns faster on Humanoid with a similar final return. With the distributional critic, iBOLT + IDAC critic achieves higher returns than IDAC on Hopper, HalfCheetah, and Ant, and is on par on Walker2d and Humanoid. The forward-KL actor therefore improves over the reverse-KL semi-implicit actor under either critic, and the distributional critic gives a separate gain that applies to both actors.

[Q4: Comparison with CrossQ]

iBOLT aims to learn a multimodal policy efficiently, so our experiments mainly compare it with methods that learn expressive policies (SQL, MEow, DIPO, and MFPO). CrossQ instead improves the critic and keeps a unimodal Gaussian actor. It is therefore orthogonal to iBOLT, not a direct baseline, and the two can be combined.

Still, as the reviewer requested, we compared iBOLT with the CrossQ critic and the official CrossQ on Hopper and Humanoid (Appendix E.3. Figure 17, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/LearningCurve_crossq.png). Together with SAC and iBOLT from Section 5.2, this gives a two-by-two comparison of the critic and the actor at 1M steps:

CriticActorHopperHumanoid
Standard twin critictanh-Gaussian (SAC)2789±7282789 \pm 7284856±2824856 \pm 282
Standard twin criticiBOLT3454±1413454 \pm 1415116±1365116 \pm 136
CrossQtanh-Gaussian (CrossQ)2533±4692533 \pm 4699683±1869683 \pm 186
CrossQiBOLT2345±7102345 \pm 7109566±1389566 \pm 138

[Edit: In the previous version of this comment, the Hopper and Humanoid columns were swapped in the two CrossQ rows. They are corrected above. The learning curves in Figure 17 of the revision were not affected.]

The returns are determined mainly by the critic. CrossQ raises the Humanoid return and lowers the Hopper return for both actors, and with the same critic iBOLT matches the tanh-Gaussian actor.

What iBOLT adds under the same critic is a multimodal policy at the same one-step inference cost. Across 100 randomly selected visited states, the iBOLT policy trained with the CrossQ critic has two to three persistent modes on average, while the tanh-Gaussian policy has a single mode (Appendix E.3, Tables 8 and 9, Figure 16, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/hopper_crossq_multimodality.png, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/humanoid_crossq_multimodality.png).

The reported results are difficult to reconcile with the original CrossQ paper, which demonstrates returns exceeding 10,000 on Humanoid within 1M training frames. This discrepancy raises concerns about the correctness of your implementation and experimental setup. More importantly, the reported return of 9,566 on Hopper also appears unusually high. Could the authors carefully verify their implementation, evaluation protocol, and reported results, and provide an explanation for these discrepancies? (Your table is also different from the learning curves.) Based on these concerns, I maintain my initial score.

Furthermore, I consider CrossQ an essential baseline, particularly given your claim that multimodal generative policies outperform unimodal Gaussian policies. Since CrossQ provides an important point of comparison for evaluating this claim, its omission is difficult to justify. I strongly encourage the authors to reconsider this baseline and include a thorough comparison.

Reviewer_8dm6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents iBOLT, that learns a one-step policy represented as a continuous mixture of latent-conditioned truncated Gaussians. During training, it constructs a finite mixture, samples candidate actions, and uses density-corrected, self-normalized importance weights to fit a critic-induced Boltzmann target by weighted likelihood maximization. Theoretical analysis establishes a finite-mixture upper bound on forward KL and characterizes its leading approximation penalty. Experiments examine fixed multimodal targets, multi-goal navigation, and five MuJoCo tasks along with comparisons against a few chosen baselines.

Critical strengths
  1. The matched-architecture comparison supports the importance of the training objective, not expressivity alone. An expressive policy can still lose useful modes through its optimization objective. The comparison supports iBOLT's finite-sample, density-corrected fitting procedure as a way to use that representational capacity, rather than attributing improvement simply to a larger policy class. The contribution is this demonstration within the proposed method, not discovery of forward KL's already-known mode-covering tendency.
  • With the same architecture and paired initializations, the paper reports broader mode coverage from forward-KL training than reverse-KL in an illustrative toy domain. (Figure 2)
  • GMM40 adds quantitative support beyond the illustrative targets: iBOLT covers all 40 components with the lowest MMD compared to the chosen baselines. (Figure 3 and Table 1)
  1. Sparse-reward multi-goal PointMaze provides evidence that mode coverage extends to behavior with a learned critic. This connects the target-fitting result to distinct successful trajectories under sparse feedback, making a stronger case for behavioral diversity than density matching alone.
  • In 500 evaluation rollouts per method, the paper reports that iBOLT alone among the six compared methods reaches all four goals in Medium and all eight goals in Hard. (Figure 5)
  • Competitive performance of iBOLT in terms of returns on MuJoCo and reported one-step inference latency provide supporting evidence that this approach need not sacrifice practical control efficiency. (Figure 6 and Table 2)
Critical weaknesses
  1. The proposal used in the strongest target-recovery experiment is insufficiently specified. The numerical target-recovery evidence is strong, but its correctness and mechanistic interpretation require a few distinctions.
  • The derivation samples from a finite mixture and divides importance weights by that proposal's density. GMM40 instead lists policy log-standard-deviation bounds [−5, −3.5] and a proposal standard-deviation floor of 0.05. Since exp⁡(−3.5)≈0.0302\exp(-3.5) \approx 0.0302, these appear to describe different sampling and policy scales. (Section 4.3, lines 242–259; Table 3, lines 985–997)
  • A broader proposal is valid if the importance denominator uses its actual density. The paper should identify the sampling density, correction density, and fitted density separately; their mismatch is not established from the text alone.
  • The sample-count ablation varies both latent count NN and candidate count MM together. At M=1M=1, the normalized weight is identically one, so the actor update is target-independent. Under Algorithm 1's exact self-sampling rule, its expected raw gradient is zero by the score identity; this need not hold for a distinct broadened proposal. Moreover, N=1N=1 does not make the deployed continuous mixture a single Gaussian. Poor performance therefore cannot isolate insufficient expressivity or validate the claimed specialization mechanism. Equation 11, lines 254–259; Algorithm 1, lines 796–804; Appendix D.3, lines 1273–1289.
  1. The efficiency and exploration claims lack a direct mixture-policy comparison and broader validation.
  • The paper discusses IDAC's semi-implicit Gaussian-mixture actor, but no discussion or comparison aginst PMOE [1], Gamid [2] or AEAP [3] has been presented. Gamid uses Gaussian-mixture actors with deterministic policy gradients and a diversity objective; AEAP uses a TD3-style actor ensemble with adaptive pruning, inducing mixture-like exploration through actor selection and Gaussian noise. They are relevant practical alternatives. These comparisons matter because non-iterative mixture or ensemble policies offer another route to expressive action generation without diffusion-style sampling. Matched return, coverage, training cost, and inference measurements would test whether iBOLT's particular construction improves that trade-off; a component/ensemble-size sweep would avoid privileging one arbitrary capacity.
  • No evaluation on complex Fetch or Adroit (sparse-reward) environments is reported in the paper which limits generalizability.
  • iBOLT selects temperatures from seven values per task. Equivalent baseline tuning budgets and independent selection seeds are not established. Framework differences are disclosed, but synchronization, compilation treatment, and timing repetitions also need specification. (Appendix C).

[1] Ren, Jie, et al. "Probabilistic mixture-of-experts for efficient deep reinforcement learning." arXiv preprint arXiv:2104.09122 (2021).

[2] Dey, Sheelabhadra, and Guni Sharon. "Comparing deterministic and soft policy gradients for optimizing Gaussian mixture actors." Transactions on Machine Learning Research (2024).

[3] Zhang, Wei, and Guni Sharon. "AEAP: A Reinforcement Learning Actor Ensemble Algorithm with Adaptive Pruning." Transactions on Machine Learning Research (2025).

Questions for authors
    • How do MuJoCo returns change when sampling the full mixture? What evaluation rule is used for each baseline, and does Table 2 measure the same action-generation procedure used for the return comparison?
  1. How were temperatures selected relative to reported seeds, what tuning budgets were given to baselines, and how were GPU synchronization, compilation, and warm-up handled?
  2. How does the method differ from Gamid and AEAP in terms of return performance and computational cost? Does its return-cost advantage persist against a suitable mixture/ensemble baseline with comparable tuning and a component-count sweep?
  3. Can the authors also report reward-discovery frequency, success, and goal coverage across independent training seeds on the sparse PointMaze tasks? These measurements would help distinguish fitting multiple known high-value regions from discovering useful regions with an initially inaccurate critic.
2 comments

Response to Reviewer 8dm6

We thank Reviewer 8dm6 for the detailed review and for the positive assessment of the target-recovery results and the expressivity induced by our learning objective. To address the concerns about the specification of the GMM40 proposal and the validation of our efficiency and exploration claims, we have made the following changes in the revision.

[W1-1,2, Q1 : GMM40 proposal specification]

Thank you for catching this. The “proposal standard-deviation floor” row in Table 3 was left over from an earlier configuration and was not used for the reported results. We removed it in the revision.

In the reported GMM40 runs, the sampling density qq in Eq. (10), the density qq in the denominator of the importance weights in Eq. (11), and the density π^θ,N\hat \pi_{\theta, N} whose log-likelihood is maximized in Eq. (14) are all the same finite mixture defined in Eq. (7). Thus, the reported runs follow the derivation in Section 4.3 and Algorithm 1 exactly, with no floor applied to the proposal.

For reference, we also ran GMM40 with the proposal qq broadened with a standard-deviation floor of 0.05. This broadened mixture is used for both sampling and the importance-weight denominator, and π^θ,N\hat \pi_{\theta, N} (without the floor) is used only in Eq. (14). Even though the proposal is no longer the same as the fitted policy mixture, the learned policy still covers all 40 modes in every seed, with a similar MMD.

Proposal standard-deviation floorMMD ↓Mode Coverage ↑
Disabled (paper)0.0260 ± 0.005040/40
0.050.0276 ± 0.005740/40

[W1-3: Degenerate Importance Weighting at M=1]

Thank you for pointing this out. We agree that the poor performance at N=M=1N=M=1 should not be attributed to insufficient policy expressivity. As you pointed out, when M=1M=1, the normalized importance weight in Eq. (11) is identically one regardless of the QQ-values, so the actor update contains no target-dependent learning signal. We have revised Appendix D.3 to explain the N=M=1N=M=1 result accordingly and removed the previous single-Gaussian interpretation.

To further clarify the roles of the two sampling budgets, we added an independent N × M ablation in Appendix D.3 (https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/gmm40_nm_grid.png). We evaluate all 36 combinations of N, M in (1, 4, 16, 64, 256, 512), each for 100K updates over five seeds, without a proposal floor. N controls the finite-mixture approximation, whereas M controls the number of reference actions used to estimate the weighted objective.

Both budgets matter. At fixed N = 64, increasing M from 4 to 256 improves mean coverage from 13.4 to 34.0 modes. At fixed M = 64, increasing N from 16 to 256 improves coverage from 19.0 to 39.0 modes. With N = M = 256, all five seeds recover 40/40 modes, with MMD 0.0238 ± 0.0111 in this additional experiment. Conversely, M = 1 yields negligible coverage across all tested N, consistent with the normalized weight becoming identically one and eliminating target-dependent weighting. These results complement the tied-N = M ablation and show that allocating sufficient samples to both mixture approximation and importance-weight estimation supports reliable recovery of all 40 modes within 100K updates.

[W2-1, Q3: Comparison with mixture and ensemble actors (PMOE, Gamid, AEAP)]

Thank you for pointing us to these methods. We added a discussion of PMOE [Ren et al., 2021], Gamid-PG [Dey & Sharon, 2024], and AEAP [Zhang & Sharon, 2025] to Section 2 of the revision (highlighted in blue). These methods use a finite mixture or ensemble of actors, whereas iBOLT uses a continuous mixture over a latent variable, so the number of modes is not fixed in advance. PMOE is available only as an arxiv preprint, so we discuss it in Section 2 and compare experimentally with Gamid-PG and AEAP (Appendix E.2, Figure 15, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/MuJoCo_MixturePolicies.png).

We run Gamid-PG from its official code and AEAP from its official code with the hyperparameters reported in its paper, on the same five MuJoCo tasks. As shown in Figure 15 of the revision, iBOLT outperforms both in terms of returns on most tasks (Hopper, Walker2d, and Ant) and is on par with the better of the two on the rest. For inference cost, we implement both action-selection rules in the same JAX harness as iBOLT: Gamid-PG (N=4N=4) and AEAP with seven active actors add about 15% to the per-action latency of iBOLT (+0.031 ms and +0.033 ms on 0.204 ms, Appendix E.2), and AEAP matches iBOLT once it has pruned to a single actor.

[Edit: Updated figure, table, and appendix references to match the revised manuscript. The reported results are unchanged.]

[W2-2 Complex environment experiment - Fetch]

We have added FetchPush, FetchPickAndPlace, and FetchSlide experiments with sparse rewards and HER in Appendix E.5, Figure 19; settings appear in Appendix E.6, Table 11. Both methods share network sizes, replay and normalization procedures, and interaction budgets. iBOLT makes faster early progress on Push and achieves competitive later performance on Push and Pick-and-Place, while SAC performs better on Slide.

[W2-3, Q2: Tuning budget and timing protocol]

Thank you for raising these points. All of them are now specified in Appendix C.6 of the revision.

For iBOLT, only the temperature τ\tau is tuned. It is selected per task from six values (0.05, 0.1, 0.25, 0.5, 1.0, 5.0) using two seeds (the submission listed seven by mistake). The selected value (Table 6) is then run with five independent seeds.

For the baselines, all MLP-based methods share a 2×2562 \times 256 MLP backbone to match network capacity. SAC, TD3, and DIPO use their published hyperparameters. For MFPO and SQL, whose official settings use a different backbone, we additionally searched one hyperparameter for each, using two separate seeds: the target-entropy coefficient for MFPO over (1/4, 1/2, 1, 2, 4) ×\times and the reward scale for SQL over (1/3, 2/3, 1, 2, 3)×\times the default of the official implementation. The default (1×1 \times) was best in both cases, so the reported results use it. MEow keeps its own flow-based architecture and official hyperparameters.

For timing, compilation and graph construction are excluded from both the training time and the inference latency. Each action is timed from a NumPy observation to a NumPy action, so device synchronization is included in the same way for every framework. Latencies are averaged over 5 seeds. For each seed, we first generate 300 actions as warm-up and then time the next 3,000 actions.

[Q1: Returns under full-mixture sampling, evaluation rules, and Table 2]

As the reviewer requested, we added this evaluation in Appendix E.1 (Figure 13, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/MuJoCo_zEval.png). We evaluate the same trained policies under three rules: (i) z=0z=0, which executes tanh⁡(μθ(s,0))\tanh(\mu_\theta(s,0)) and is used in all other MuJoCo results, (ii) z∼N(0,I)z\sim\mathcal{N}(0,I) sampled at every step, executing the component center tanh⁡(μθ(s,z))\tanh(\mu_\theta(s,z)), and (iii) the full mixture, where the action is also sampled from kθ(⋅∣s,z)k_\theta(\cdot\mid s,z) in Eq. (2).

Sampling zz alone gives nearly the same returns as z=0z=0, while sampling the full mixture lowers the returns on some tasks. This is the usual gap between stochastic and deterministic evaluation, so we keep z=0z=0 as the default evaluation rule, following the common practice of deterministic evaluation on MuJoCo benchmarks. The z=0z=0 rule is used only at evaluation, and training data is collected with actions sampled from the full mixture, as in Algorithm 1.

Each baseline uses the evaluation rule of its official implementation: SAC uses tanh⁡\tanh of the Gaussian mean, TD3 its deterministic actor, SQL its stochastic sampler with a new latent at every step, MEow deterministic inference through the inverse flow with z=0z=0, DIPO its 100-step reverse diffusion, and MFPO the best of 10 MeanFlow candidates scored by the critic.

Yes, Table 2 measures the same action-generation procedure used for the return comparison, for every method. For iBOLT this is z=0z=0, and sampling the full mixture increases the latency only from 0.204 ms to 0.225 ms (Appendix E.1).

[ Q4: PointMaze reward discovery and coverage ]

As the reviewer suggested, we report reward discovery, success, and goal coverage for the PointMaze experiment over five seeds in Appendix E.4 (Figure 18, https://anonymous.4open.science/r/iBOLT-243F/rebuttal_png/pointmaze_reb.png). The success rates and the numbers of goals reached per seed at 1M steps are:

MethodMedium success (%)Medium goals / 4Hard success (%)Hard goals / 8
iBOLT100.0 ± 0.04.0 ± 0.0100.0 ± 0.08.0 ± 0.0
SAC100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0
TD360.0 ± 54.80.6 ± 0.580.0 ± 44.70.8 ± 0.4
MEow100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0
SQL5.1 ± 5.91.6 ± 0.920.2 ± 13.64.8 ± 2.2
MFPO91.5 ± 15.41.0 ± 0.035.6 ± 48.90.4 ± 0.5
DIPO100.0 ± 0.01.0 ± 0.0100.0 ± 0.01.0 ± 0.0

[Edit: Updated figure, table, and appendix references to match the revised manuscript. The reported results are unchanged.]

Comments on the paper

1 comment

Summary of Our Responses and Revision

Dear AC and Reviewers,

We thank the AC and all reviewers for their time and constructive feedback. To help the ongoing discussion, we briefly summarize our contribution and the concerns we have addressed in the revision so far.

Contribution. iBOLT asks whether an expressive multimodal Boltzmann policy can be learned in online RL without multi-step generation, auxiliary score/velocity networks, or invertible architectures. It combines (1) an expressive semi-implicit policy with one-step action generation, (2) a forward-KL minimization derived from a finite-mixture upper bound (Proposition 1) that achieves broader mode coverage than reverse-KL training with the same architecture, and (3) inference latency matching that of Gaussian policies with substantially lower training cost than other expressive policies.

Main concerns and how they were addressed.

  1. Comparison with semi-implicit and mixture policy baselines. (8dm6, uAXn) First, we showed that with the same semi-implicit policy class and the same critic, forward-KL training (iBOLT) matches or exceeds reverse-KL training (the actor of IDAC) on all five MuJoCo tasks, suggesting that the training objective can matter. Second, we showed that a continuous mixture from a single network matches or exceeds prior non-iterative approaches to expressive actors such as finite mixtures and actor ensembles. Against Gamid-PG and AEAP, iBOLT achieves higher returns on three of the five tasks and is on par on the other two (Appendix E.2).
  2. Does iBOLT learn multimodal policies beyond toy targets? (uAXn, ZHgi) Yes. We examine the learned action distributions on Hopper and Humanoid through visualizations and quantitative measurements in the full action space. Under the same CrossQ critic, iBOLT learns multiple persistent modes on average while achieving returns comparable to those of the predominantly unimodal tanh-Gaussian actor (see Figure 16 and Table 9 in Appendix E.3). In PointMaze, iBOLT reaches all goals in every seed, while SAC, MEow, and DIPO reach a single goal per seed (Appendix E.4).
  3. Evaluation rule and tuning protocol. (8dm6, uAXn, ZHgi) We added returns under sampled latents and under full-mixture sampling (Appendix E.1), and specified the tuning budgets, baseline evaluation rules, and timing protocol (Appendix C.6).
  4. Inconsistent GMM40 setting in Table 3. (8dm6, uAXn) Table 3 listed a proposal standard-deviation floor that did not match the sampling procedure of Algorithm 1. This entry was left over from an earlier configuration and was not used in the reported runs, so we removed it. The reported runs follow Algorithm 1 exactly, and 40/40 mode coverage also holds when the floor is enabled.

We hope this summary is helpful, and we would be glad to answer any remaining questions during the rest of the discussion period.

Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.