- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents iBOLT (not a 1:1 acronym for what it actually does), an approach that builds on recent advances in 1-step generative policies, and frames it in two interesting ways. First, it creates a Boltzmann target that more naturally captures multi-modal action distributions (or state-action value distributions). Second, it creates a set of loss functions based on importance sampling (and the aforementioned Boltzmann style target) that re-cast the problem as an implicit policy optimization one. The paper -- cleanly, in my opinion -- describes this narrative both mathematically and algorithmically, and then proceeds to an empirical evaluation on robotics or navigation style tasks that are either (a) know to be multi-modal and/or (b) are standard in the RL literature.
Critical strengths
-
The paper is organized well and the prose is generally concise and clear.
-
The problem is well-motivated in terms of both established and emerging methods in ML broadly and RL specifically. For example, it melds advances in multi-model representations (and doing it in 1 step) which have gained popularity relatively recently, along with now-class approaches that use implicit parameterizations from the RL literature.
-
The story in section 4 is nice. It proceeds in a clear way from well-known "tricks" and methods in order to derive the critic loss function and ultimately the policy extraction method in (15). There derivation is sound, and the subsequent discussion (at the end of section 4) brings things to a close in a concise, conceptual way.
-
The questions that kick off section 5 do a nice job of clearly identifying and driving the agenda of the paper. These questions are also essentially the contributions; as such they are mentioned in section 1. However, I suggest the authors consider simply adding these questions to the intro; this is not a requirement, just a suggestion that would perhaps only take a bit of light editing.
Critical weaknesses
-
The evaluations are largely on "toy" problems. These are, however, common in RL literature. The issue is that claims about multi-modality, expressitivity, etc are limited (at least empirically).
-
The experiments do well to try to evaluate the paper's central claims. However, I believe the paper is not rigorous enough (or, perhaps, descriptive enough) regarding the design choices of the architectures. This includes choices for iBOLT itself but more importantly surrounds the various baselines and their implementation.
Questions for authors
-
The TD loss in equation (4) is basically standard, but for clarity it would help to be explicit about how to get from the two sets of parameters learned in (4) to the Boltzmann target in (5), which only contains one Q network. It is in the pseudocode in the appendix, perhaps say how this target update works in one conceptual sentence.
-
In section 5, the baseline and iBolt architectures use the same hyperparameters, especially when the objective of an experiment is to isolate things that are outside of the architecture. However, some of the hyperparameters seem to be selected arbitrarily (e.g. 128 latent samples and candidate actions). Is this number optimized? Would a different number, say, benefit reverse-KL training? It is difficult to know without parameter sweeps.
-
Similarly, for the GMM40 multimodal recovery study, it is unclear how each method has been optimized (or not) in terms of hyperparameters? Do the baselines use off-the-shelf designs? Has iBOLT been tuned? For some of the Mujoco tasks, baseline architectural details are shown in the appendix. However, these are all the "same" or similar. I assume this is to isolate for compute and attempt to get 1:1 comparisons in terms of computational resources. However, these hyperparameters might happen to be optimal for iBOLT and severely suboptimal for the baselines.
-
Why isn't MEoW included in the first study? Or more generally, why aren't the baselines the same for every experiment? It does not seem like there is anything fundamentally constraining, either in terms of the RL algorithms or the environments, to prevent this.
-
What are the error bars -- or -- for Table 2?
4 comments
[Q3: Hyperparameters and fairness of the comparisons]
For GMM40, each baseline keeps the architecture and sampling procedure of its official implementation, and all methods share the target ( with temperature 1), optimizer, learning rate, batch size, and number of updates (Table 3). iBOLT was tuned on this task: was chosen from the sweep in Appendix D.3 (Table 7), and the scale bounds and initialization in Table 3 are specific to GMM40. The baselines use the same hyperparameters as in the MuJoCo experiments, since tuning them on GMM40 did not change the results significantly.
For MuJoCo, the shared settings in Table 5 are the standard settings of SAC and TD3, which iBOLT adopts. They are therefore not values tuned for iBOLT. For iBOLT, only the temperature is tuned, from six values with two seeds, and is fixed across tasks without a search. SAC, TD3, and DIPO use their published hyperparameters, and MEow keeps its own flow-based architecture and official hyperparameters. For MFPO and SQL, whose official settings use a different backbone, we additionally searched the target-entropy coefficient over (1/4, 1/2, 1, 2, 4) and the reward scale over (1/3, 2/3, 1, 2, 3) the default of the official implementation. The default was best in both cases.
[Q4: Baseline selection across experiments]
The MuJoCo comparison includes all six baselines (SAC, TD3, MEow, SQL, MFPO, DIPO). The other two experiments each omitted some of them, and we have now added the missing ones.
In GMM40, TD3 is not applicable because its policy is deterministic and cannot represent a target distribution. MEow was not included in the original study. We have now added it under the same setting as the other baselines, using its released architecture and objective (Figure 3):
| Method | MMD ↓ | Mode coverage ↑, out of 40 |
|---|---|---|
| iBOLT | 0.0260 ± 0.0050 | 40.0 ± 0.0 |
| MEow | 0.3029 ± 0.1019 | 24.0 ± 10.15 |
In PointMaze, DIPO was not included in the Figure 5. We now report it on both the Medium and Hard mazes:
| Method | Medium success (%) | Medium goals / 4 | Hard success (%) | Hard goals / 8 |
|---|---|---|---|---|
| iBOLT | 100.0 ± 0.0 | 4.0 ± 0.0 | 100.0 ± 0.0 | 8.0 ± 0.0 |
| SAC | 100.0 ± 0.0 | 1.0 ± 0.0 | 100.0 ± 0.0 | 1.0 ± 0.0 |
| TD3 | 60.0 ± 54.8 | 0.6 ± 0.5 | 80.0 ± 44.7 | 0.8 ± 0.4 |
| MEow | 100.0 ± 0.0 | 1.0 ± 0.0 | 100.0 ± 0.0 | 1.0 ± 0.0 |
| SQL | 5.1 ± 5.9 | 1.6 ± 0.9 | 20.2 ± 13.6 | 4.8 ± 2.2 |
| MFPO | 91.5 ± 15.4 | 1.0 ± 0.0 | 35.6 ± 48.9 | 0.4 ± 0.5 |
| DIPO | 100.0 ± 0.0 | 1.0 ± 0.0 | 100.0 ± 0.0 | 1.0 ± 0.0 |
DIPO reaches 100% success on both mazes with a single goal per seed, as SAC and MEow do, while iBOLT reaches all four goals in Medium and all eight goals in Hard.
[Q5 : Error bars in Table 2]
We added them in the revision. Table 2 now reports the latency as mean ± standard deviation over 5 seeds for every method (for example, ms for iBOLT). For each seed, we first generate 300 actions as warm-up and then measure the latency of the next 3,000 actions. The full measurement protocol is given in Appendix C.6.
I thank the authors for their thorough rebuttal. The new results in Appendix E are indeed quite interesting.
I hope the authors feel that the paper is now a bit more clear and precise, and overall "better". It seems to me that it is.
I stand by my position that current standard benchmarks are probably insufficient for empirical arguments regarding claims of complexity, multi-modality, etc. This is not a criticism of the authors or the paper; it is simply a statement about the state of things.
I am not asking for another revision; however, I would like to note that many of the claims are about multi-modality, and the main body text has been improved to reflect this. I am not asking the authors to include (for example) App E.3 in the main body, but I suggest in the final camera-ready version the main body at least directs the reader to this and other details (as it already does for most other parts of the appendix).
Dear Reviewer ZHgi,
Thank you again for your thoughtful follow-up and encouraging comments. Your constructive feedback throughout the review process has helped us improve the clarity and precision of our paper.
Following your suggestions, we clarified the construction of the Boltzmann target, expanded the sample-count ablations and baseline comparisons, and provided more detailed experimental settings and latency statistics. Together with the additional multimodality analyses, these revisions have helped us present a clearer and more comprehensive evaluation of iBOLT. We also appreciate your observation that current standard benchmarks have limitations in fully assessing multimodal policy representations, and agree that this could be an important direction for future research.
Regarding your suggestion on the presentation of these results, should we have the opportunity to prepare a final camera ready version of the manuscript, we will further revise the main text to explicitly direct readers to the multimodality analysis in Appendix E.3 and the other newly added results in Appendix E.
Thank you once again for your time and thoughtful review.
Best regards, Authors
Response to Reviewer ZHgi
We greatly appreciate the reviewer's thoughtful and constructive feedback. We are especially grateful for the helpful suggestion on the Introduction: we have incorporated the suggested questions into the revised Introduction. Our responses to the remaining points are given below.
[W1: Evaluation beyond toy problems]
In the revision, we provide additional evidence that the learned policies are multimodal on standard continuous control benchmarks. On Hopper and Humanoid, we measure the number of persistent modes of the learned action distributions in the full action space, over 100 randomly selected visited states. iBOLT has two to three modes on average, while the tanh-Gaussian policy trained with the same critic has a single mode (Appendix E.3, Tables 8 and 9, Figure 16).
[W2: Design choices for iBOLT and the baselines]
For iBOLT, the main components follow from the derivation in Section 4 and are not separate design choices. The semi-implicit actor is introduced for expressivity, the finite-mixture approximation handles its intractable marginal density and gives an upper bound on the forward KL (Proposition 1), and self-normalized importance sampling gives a consistent estimate of the expectation under the intractable Boltzmann target. The remaining implementation choices are listed in Appendix C.4 (Table 4) and ablated in Appendix D: the temperature (D.1), the exploration noise (D.2), and the numbers of latent and reference samples (D.3).
For the baselines, we selected a representative method from each family of expressive policies for online RL: diffusion (DIPO), flow matching (MFPO), normalizing flow (MEow), and implicit sampling (SQL), together with the standard SAC and TD3. Each is run from its official implementation (Appendix C.1), and Appendix C.6 (Table 5) lists their architectures and hyperparameters, where all MLP-based methods share a 2×256 MLP backbone to match network capacity. In the revision, we also added the tuning procedure for iBOLT and the baselines, the evaluation rule of each baseline, and the timing protocol (Appendix C.6).
[Q1: From the two critics in Eq. (4) to the Boltzmann target in Eq. (5)]
Thank you for pointing this out. We train two critics Qϕ1 and Qϕ2 with the loss in Eq. (4), where the minimum over the two target critics gives the clipped double-Q target. For policy learning, the Boltzmann target in Eq. (5) uses the mean of the two online critics, Qˉϕ(s,a)=21(Qϕ1(s,a)+Qϕ2(s,a)). The target critics are updated as an exponential moving average of the online critics (line 15 of Algorithm 1). We added this explanation to Section 4.2 and now write Eq. (5) with Qˉϕ.
[Q2: Sweep over N and M for the forward- and reverse-KL experiment of Figure 2]
The value N=M=128 is a default and was not optimized. To see its effect, we ran the full grid N,M∈{32,128,512} for both objectives on the two targets of Figure 2: the three-mode Gaussian mixture (Figure 2a-b) and the spike-and-ramp target (Figure 2c-d), with three seeds per setting. The table reports the mean total variation (TV) distance to the target and, in brackets, the number of seeds that miss at least one mode.
Our forward-KL training recovers all modes in every seed once M≥128. In contrast, reverse-KL training misses a mode in 50 of 54 runs, including every run on the three-mode Gaussian mixture target. The conclusion of Figure 2 therefore does not depend on the specific choice of 128. For reverse-KL training, N and M set the number of sampled latents and actions per group, and the marginal density is estimated separately with L=217 latent samples.
[Edit: Updated figure, table, and appendix references to match the revised manuscript. The reported results are unchanged.]