- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the MALA-Guided Mirror Descent (MGMD) method. MGMD approximates the KL-regularized policy mirror descent (PMD) target using annealed MALA sampling and continuously distills the improved actions back into the actor. Experiments demonstrate that MGMD approximates the PMD target more accurately than AWR in controlled multimodal bandit environments and achieves competitive online RL performance across six MuJoCo tasks.
Critical strengths
S1. The paper presents a well-defined policy improvement framework by formulating diffusion policy improvement as a sampling problem and using PMD to define a theoretically grounded target distribution.
S2. The paper demonstrates competitive empirical performance across six MuJoCo continuous-control tasks, with comparisons against SAC, DIPO, and DPMD, the latter serving as an objective-matched baseline.
Critical weaknesses
W1. Insufficient Motivation for Adopting PMD in the Introduction. The paper's introduction moves directly from the limitations of existing methods to Mirror Descent. However, the presentation would be much clearer if the introduction explained why Policy Mirror Descent (PMD) is used and how PMD provides the necessary policy improvement target mentioned earlier.
W2. The existing experiments do not sufficiently substantiate the core motivation. The paper notes in the Introduction that while existing inference-time guidance methods can generate actions with high Q-values, the lack of a well-defined policy improvement target means that subsequent policy distillation does not necessarily achieve a clear policy improvement update. However, Figure 1 primarily compares how accurately MGMD and AWR approximate the PMD target in terms of Jensen–Shannon (JS) divergence; it does not directly evaluate the aforementioned limitations of the relevant inference-time guidance methods. Although the MuJoCo experiments demonstrate competitive downstream performance for MGMD, a direct link between these results and the core motivation has not yet been established. Targeted quantitative comparisons with relevant inference-time guidance baselines would help establish whether the proposed method achieves more effective policy improvement after distillation.
W3. The computational efficiency of the proposed method is insufficiently evaluated. The paper does not report training wall-clock time, inference latency, or action sampling time. A quantitative comparison of computational costs across the methods shown in Figure 2 would help assess the practical efficiency of MGMD.
Questions for authors
Q1. Does a lower PMD target error systematically translate into better online RL performance?
Q2. Given that MGMD relies on additional inference-time computation, could the authors further evaluate its performance-compute trade-off? Specifically, how do its performance gains over baselines such as DPMD compare with the additional training and inference costs?
Thank you for your helpful and constructive review.
We have clarified the motivation for PMD, the conclusions supported by each experiment, and the computational cost of the method.
The revised introduction explains that PMD maximizes expected critic value while penalizing departure from the current policy. Its exponential-tilt solution specifies a target distribution, with a coefficient controlling the strength of the update. This motivates studying how accurately a diffusion policy can realize and subsequently distill that distribution.
Lower target error alone does not guarantee higher online returns. Critic accuracy, exploration, and distillation also affect learning. We now distinguish the experimental conclusions accordingly. The controlled bandit measures fidelity to a known PMD target, while MuJoCo evaluates the complete learning procedure. The revised bounded bandit uses 100 diffusion levels and includes K = 1. Across all tested tilt strengths, K=1 consistently achieves lower target error than reweighted score matching (RSM; equivalently, advantage-weighted regression, AWR), and this gap generally widens as the target moves farther from the base distribution. Additional MALA updates further improve accuracy at strong tilts.
We report both network-operation counts and wall-clock measurements. All methods use 100 diffusion levels, batch size B=256, one environment, and one training iteration per environment transition. We retain each method's native training rule: MGMD uses one MALA update per diffusion level, DIPO uses 20 critic-gradient action-refinement steps, and DPMD generates 32 diffusion candidates per state and updates its actor every other iteration. Each critic operation below includes the twin-Q pair.
For MGMD, each of the 100 MALA levels evaluates the guided energy and drift at both the current and proposed actions. The proposed-state evaluations required by the Metropolis correction are therefore included in the counts above. The acceptance calculation itself reuses the evaluated energies and drifts and requires no additional network evaluation. Hessian--vector and mixed-derivative products are computed through automatic-differentiation vector products without constructing Hessian matrices, and are counted separately from ordinary gradients. Energy values and input gradients share their forward evaluations.
We additionally measured complete training-loop time on one NVIDIA H100. The measurements are means over 250 synchronized iterations after compilation and warmup. They include action generation, environment interaction and resets, replay-buffer operations, data transfers, and learning updates, while excluding compilation, warmup, separate evaluation, external logging, and checkpointing.