acceptodds
Under review as a conference paper at ICLR 2027

MALA-Guided Mirror Descent

10 views

Abstract

Policy mirror descent (PMD) defines policy improvement as a simple distribution-space update: exponentially tilt the current policy toward high-value actions while remaining close to the original policy. For expressive generative policies, however, realizing this prescribed target distribution can itself be difficult. We introduce MALA-Guided Mirror Descent (MGMD), which treats policy improvement as an inference problem rather than solely as an actor-training objective. MGMD represents the diffusion policy through an energy function and uses annealed Metropolis-adjusted Langevin dynamics to traverse a sequence of value-tilted noisy distributions whose zero-noise limit is the PMD target. The resulting improved actions are used for environment interaction and Bellman targets, and are continuously distilled back into the diffusion policy with the standard denoising objective. On a controlled multimodal bandit where the exact PMD target is known, MGMD tracks the target more accurately than advantage-weighted regression as the target moves farther from the base policy. Across six MuJoCo continuous-control tasks, MGMD demonstrates consistently strong performance. These results suggest that inference-time computation can provide a persistent mechanism for policy improvement when it is directed toward a prescribed policy-improvement distribution and subsequently amortized into the policy.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Diffusion, Energy-Based Models, Generative Models, Mujoco, Robotics, Continuous Action Space

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ATbaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the MALA-Guided Mirror Descent (MGMD) method. MGMD approximates the KL-regularized policy mirror descent (PMD) target using annealed MALA sampling and continuously distills the improved actions back into the actor. Experiments demonstrate that MGMD approximates the PMD target more accurately than AWR in controlled multimodal bandit environments and achieves competitive online RL performance across six MuJoCo tasks.

Critical strengths

S1. The paper presents a well-defined policy improvement framework by formulating diffusion policy improvement as a sampling problem and using PMD to define a theoretically grounded target distribution.

S2. The paper demonstrates competitive empirical performance across six MuJoCo continuous-control tasks, with comparisons against SAC, DIPO, and DPMD, the latter serving as an objective-matched baseline.

Critical weaknesses

W1. Insufficient Motivation for Adopting PMD in the Introduction. The paper's introduction moves directly from the limitations of existing methods to Mirror Descent. However, the presentation would be much clearer if the introduction explained why Policy Mirror Descent (PMD) is used and how PMD provides the necessary policy improvement target mentioned earlier.

W2. The existing experiments do not sufficiently substantiate the core motivation. The paper notes in the Introduction that while existing inference-time guidance methods can generate actions with high Q-values, the lack of a well-defined policy improvement target means that subsequent policy distillation does not necessarily achieve a clear policy improvement update. However, Figure 1 primarily compares how accurately MGMD and AWR approximate the PMD target in terms of Jensen–Shannon (JS) divergence; it does not directly evaluate the aforementioned limitations of the relevant inference-time guidance methods. Although the MuJoCo experiments demonstrate competitive downstream performance for MGMD, a direct link between these results and the core motivation has not yet been established. Targeted quantitative comparisons with relevant inference-time guidance baselines would help establish whether the proposed method achieves more effective policy improvement after distillation.

W3. The computational efficiency of the proposed method is insufficiently evaluated. The paper does not report training wall-clock time, inference latency, or action sampling time. A quantitative comparison of computational costs across the methods shown in Figure 2 would help assess the practical efficiency of MGMD.

Questions for authors

Q1. Does a lower PMD target error systematically translate into better online RL performance?

Q2. Given that MGMD relies on additional inference-time computation, could the authors further evaluate its performance-compute trade-off? Specifically, how do its performance gains over baselines such as DPMD compare with the additional training and inference costs?

1 comment

Thank you for your helpful and constructive review.

We have clarified the motivation for PMD, the conclusions supported by each experiment, and the computational cost of the method.

The revised introduction explains that PMD maximizes expected critic value while penalizing departure from the current policy. Its exponential-tilt solution specifies a target distribution, with a coefficient controlling the strength of the update. This motivates studying how accurately a diffusion policy can realize and subsequently distill that distribution.

Lower target error alone does not guarantee higher online returns. Critic accuracy, exploration, and distillation also affect learning. We now distinguish the experimental conclusions accordingly. The controlled bandit measures fidelity to a known PMD target, while MuJoCo evaluates the complete learning procedure. The revised bounded bandit uses 100 diffusion levels and includes K = 1. Across all tested tilt strengths, K=1K=1 consistently achieves lower target error than reweighted score matching (RSM; equivalently, advantage-weighted regression, AWR), and this gap generally widens as the target moves farther from the base distribution. Additional MALA updates further improve accuracy at strong tilts.

We report both network-operation counts and wall-clock measurements. All methods use 100 diffusion levels, batch size B=256B=256, one environment, and one training iteration per environment transition. We retain each method's native training rule: MGMD uses one MALA update per diffusion level, DIPO uses 20 critic-gradient action-refinement steps, and DPMD generates 32 diffusion candidates per state and updates its actor every other iteration. Each critic operation below includes the twin-QQ pair.

Operation per training iterationMGMDDIPODPMD
Policy forward202B202B101B+100101B+1003200(B+1)+B/23200(B+1)+B/2
Policy input gradient202B202B0000
Policy parameter gradient00BBB/2B/2
Policy Hessian--vector product201B201B0000
Policy mixed-derivative productBB0000
Critic forward204B204B22B22B35B+3235B+32
Critic input gradient201B201B20B20B00
Critic parameter gradientBBBBBB

For MGMD, each of the 100 MALA levels evaluates the guided energy and drift at both the current and proposed actions. The proposed-state evaluations required by the Metropolis correction are therefore included in the counts above. The acceptance calculation itself reuses the evaluated energies and drifts and requires no additional network evaluation. Hessian--vector and mixed-derivative products are computed through automatic-differentiation vector products without constructing Hessian matrices, and are counted separately from ordinary gradients. Energy values and input gradients share their forward evaluations.

We additionally measured complete training-loop time on one NVIDIA H100. The measurements are means over 250 synchronized iterations after compilation and warmup. They include action generation, environment interaction and resets, replay-buffer operations, data transfers, and learning updates, while excluding compilation, warmup, separate evaluation, external logging, and checkpointing.

MeasurementEnvironmentMGMDDIPODPMD
Training time (ms/iteration)HalfCheetah30.89102.1313.45
Training time (ms/iteration)Humanoid37.34100.5317.75
Reviewer_MB8WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MGMD uses annealed MALA to sample from an exponential tilt of a diffusion policy defined by its critic. The sampled actions are used for interaction, Bellman targets, and policy distillation. A two-dimensional bandit with a known target distribution tests sampling error. Online control experiments cover six MuJoCo tasks and ten seeds, with ablations of sampling steps, step-size adaptation, and TD targets. The method uses a designed annealing path, and finite mixing remains an acknowledged approximation.

Critical strengths
  1. I like the decision to test sampling against a known target before examining control returns. It directly addresses whether the sampler realizes the intended policy update. The exponential-tilt construction and fixed-step MH acceptance correction appear algebraically consistent under the stated assumptions.

  2. The six-task, ten-seed evaluation is useful, with the clearest gain on Ant: 6,688 versus 5,805 for DPMD, an increase of 883 (Table 1). These scores average the per-checkpoint medians over the final ten evaluations. HalfCheetah is much closer, at 12,949 versus 12,808, and the reported 80% bootstrap intervals overlap. I read these results as competitive overall performance, with benefits that vary across tasks.

Critical weaknesses
  1. The known-target evidence does not yet settle the approximation quality of the finite sampler used in control. MuJoCo uses K=1 update per noise level and an adaptation rate of 0.2, followed by a Tweedie-based output and action clipping. The strongest bandit target-matching results use K at least 16 (Figure 1). The paper acknowledges finite mixing; fixed-kernel invariance alone does not characterize the practical output distribution. A corresponding K=1 diagnostic would help quantify the remaining approximation, without implying that the displayed MH kernel is invalid.

  2. The 50-step versus 100-step ablation addresses diffusion depth, but not the full cost comparison. Energy gradients, critic gradients, acceptance calculations, and possible higher-order derivatives can have different costs from baseline score evaluations. The methods also differ in energy parameterization and their noise schedules. Without measured time or a more closely matched control, I cannot tell how much gain comes from the sampler itself at comparable compute.

Questions for authors
  1. On the existing bandit, how well does K=1 with the practical adaptation and terminal transformation match its stated target? If action clipping is tested, please define the corresponding bounded reference distribution, since the original bandit is unbounded.

  2. How are energy and critic evaluations, acceptance calculations, and first- or higher-order derivatives counted? Could you give the differentiation/caching choices and any available per-action or training-time measurements for the 50-step and 100-step variants?

I will further consider my score in light of the other reviewers’ comments and whether the authors’ response adequately addresses my concerns.

1 comment

Thank you for your helpful and constructive review.

We have added K = 1 target-matching results with terminal processing and bounded actions, together with computational accounting and measured runtimes.

The revised bandit uses 100 diffusion levels and defines both the base distribution and the tilted target on [−1,1]2[-1,1]^2. The practical terminal action transformation is included: at the final diffusion level, we apply the DDPM posterior-mean denoising update and clip the resulting final actions to [−1,1]2[-1,1]^2. Target error is measured using one million final actions after terminal denoising and clipping. Tweedie estimates passed to the advantage function are clipped to [−3,3]2[-3,3]^2, matching the MuJoCo convention. The revised figure includes K∈{1,2,4,8}K\in\{1,2,4,8\}. Across all tested tilt strengths, K=1K=1 consistently achieves lower target error than reweighted score matching (RSM; equivalently, advantage-weighted regression, AWR), and this gap generally widens as the target moves farther from the base distribution. Additional MALA updates further improve accuracy at strong tilts.

The bandit contains a single stationary pbase→ptiltp_{\mathrm{base}}\to p_{\mathrm{tilt}} update, so we tune and fix the step size when measuring target-approximation error. MuJoCo instead performs a sequence of such updates as the actor, critic, and visited states evolve, and the appropriate step size may change across updates. We therefore increase the per-level step size when its actual acceptance rate exceeds the target acceptance rate and decrease it when the actual acceptance rate falls below the target. The resulting acceptance rates remain near the prescribed target during training, and Figure 4 compares fixed and adaptive step sizes, showing a benefit from adaptation. The MuJoCo step-size adaptation is designed to track a sequence of evolving policy updates and is therefore not used for this single stationary bandit update.

We report both network-operation counts and wall-clock measurements. All methods use 100 diffusion levels, batch size B=256B=256, one environment, and one training iteration per environment transition. We retain each method's native training rule: MGMD uses one MALA update per diffusion level, DIPO uses 20 critic-gradient action-refinement steps, and DPMD generates 32 diffusion candidates per state and updates its actor every other iteration. Each critic operation below includes the twin-QQ pair.

Operation per training iterationMGMDDIPODPMD
Policy forward202B202B101B+100101B+1003200(B+1)+B/23200(B+1)+B/2
Policy input gradient202B202B0000
Policy parameter gradient00BBB/2B/2
Policy Hessian--vector product201B201B0000
Policy mixed-derivative productBB0000
Critic forward204B204B22B22B35B+3235B+32
Critic input gradient201B201B20B20B00
Critic parameter gradientBBBBBB

For MGMD, each of the 100 MALA levels evaluates the guided energy and drift at both the current and proposed actions. The proposed-state evaluations required by the Metropolis correction are therefore included in the counts above. The acceptance calculation itself reuses the evaluated energies and drifts and requires no additional network evaluation. Hessian--vector and mixed-derivative products are computed through automatic-differentiation vector products without constructing Hessian matrices, and are counted separately from ordinary gradients. Energy values and input gradients share their forward evaluations.

We additionally measured complete training-loop time on one NVIDIA H100. The measurements are means over 250 synchronized iterations after compilation and warmup. They include action generation, environment interaction and resets, replay-buffer operations, data transfers, and learning updates, while excluding compilation, warmup, separate evaluation, external logging, and checkpointing.

MeasurementEnvironmentMGMDDIPODPMD
Training time (ms/iteration)HalfCheetah30.89102.1313.45
Training time (ms/iteration)Humanoid37.34100.5317.75
Reviewer_jA3JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MALA-Guided Mirror Descent (MGMD) for online reinforcement learning with diffusion policies. MGMD formulates policy improvement as an inference problem and it parameterizes the diffusion policy using an energy function and employs annealed Metropolis-adjusted Langevin dynamics to sample from a sequence of critic-guided distributions whose limiting endpoint corresponds to the KL-regularized PMD target.

The resulting guided actions are used for environment interaction, critic Bellman targets, and continuous policy distillation through standard denoising score matching. The paper evaluates the method on a controlled multimodal bandit with a known PMD target and six MuJoCo continuous-control environments. The results demonstrate improved target approximation in the bandit setting and competitive online RL performance.

Critical strengths

The perspective on policy improvement by explicitly constructing a KL-regularized PMD target and approximately sampling from it through annealed MALA is interesting. The framework provides a conceptually meaningful connection between policy mirror descent, energy-based diffusion models, and inference-time computation.

The paper evaluates the proposed method on six MuJoCo continuous-control tasks with ten random seeds and includes targeted ablations. MGMD achieves competitive performance across all six environments, with particularly encouraging results on Ant.

Critical weaknesses
  1. The connection between the theoretical PMD target and the policy implemented in the main experiments is insufficiently established.

Section 4.1 shows that, for a fixed step size, each MALA transition has the corresponding intermediate distribution as its invariant distribution. However, this does not guarantee that a finite number of transitions produces samples close to that distribution. In particular, the MuJoCo experiments use only K=1 MALA update per diffusion level (Appendix B.2), whereas the strongest target-matching results in Figure 1 are obtained with substantially larger values of K. The practical sampler also applies a terminal Tweedie-based transformation and action clipping, which introduce additional discrepancies from the idealized target.

Furthermore, the PMD update is defined using the value function of the reference policy, whereas the Bellman targets in Section 4.2 use actions sampled from the guided policy. Consequently, the learned critic generally evaluates the guided policy rather than the base diffusion policy. This design may be reasonable, but the relationship between the resulting algorithm and the stated PMD operator requires further clarification.

These issues matter because the central motivation is not merely to generate high-value actions, but to realize a prescribed policy-improvement distribution. The current experiments do not establish that the actual online sampler achieves this objective with sufficient accuracy.

  1. The experimental comparisons do not adequately establish the benefits of the proposed mechanism relative to its computational cost.

MGMD requires additional energy evaluations, critic guidance, Metropolis acceptance calculations, and potentially higher-order derivatives. Although Figure 3 studies different numbers of diffusion steps, matching the number of score evaluations does not necessarily imply comparable computational cost. The paper does not report training wall-clock time or action-sampling latency, making it difficult to assess the practical trade-off between performance and computation.

Moreover, while the paper discusses recent inference-time critic-guidance methods such as CGPO, it does not include a direct experimental comparison with such methods. This limits the evidence that the explicit PMD target and Metropolis correction provide benefits beyond guided action generation and distillation. The concern is particularly relevant because performance gains over DPMD are relatively small on several environments, with overlapping reported confidence intervals.

Without more direct evidence isolating the proposed mechanism or demonstrating its advantages under comparable computational budgets, the significance of the empirical improvement remains uncertain.

Questions for authors
  1. Could the authors evaluate the target approximation error in the controlled bandit using K=1, consistent with the MuJoCo implementation? In particular, how does the approximation error change when including the practical step-size adaptation and terminal action transformation?

  2. Consistency between policy improvement and critic evaluation.

The theoretical PMD update uses the value function of the reference policy, whereas Algorithm 2 constructs Bellman targets using actions sampled from the guided policy.

Could the authors clarify which policy the critic is intended to evaluate and how this choice relates to the PMD update when the distilled actor differs from the guided sampling distribution?

  1. Computational cost and the contribution of Metropolis correction.

Could the authors report training wall-clock time and per-action inference latency relative to the diffusion-policy baselines? In particular, how are the additional energy evaluations, critic gradients, and higher-order derivatives accounted for?

1 comment

Thank you for the constructive review. We have revised the bandit experiment, clarified critic evaluation, and added computational measurements.

Bandit and step sizes. The revised bandit uses 100 diffusion levels, with both base and tilted distributions on [−1,1]2[-1,1]^2. MGMD applies terminal DDPM posterior-mean denoising and clips final actions to [−1,1]2[-1,1]^2. Target error (Jensen-Shannon divergence) is measured using one million returned actions per method. Tweedie estimates passed to the advantage function are clipped to [−3,3]2[-3,3]^2, matching MuJoCo. The revised figure includes K∈{1,2,4,8}K\in\{1,2,4,8\}. At every tested tilt, K=1K=1 has lower target error than reweighted score matching (RSM; equivalent here to AWR), with a generally widening gap as the target moves farther from the base. Additional MALA updates improve accuracy at strong tilts.

The bandit measures a single stationary update, so we tune and fix its step size. In MuJoCo, the actor, critic, and visited states evolve across updates. We increase each level's step size when acceptance exceeds the target and decrease it when acceptance falls below it. Acceptance remains near the target; Figure 4 shows reduced sensitivity to step-size tuning.

Critic evaluation and PMD. The critic is trained to evaluate the finite guided-sampler policy. Actions from that sampler provide both the actor's distillation targets and the critic's Bellman-target actions.

In an idealized outer iteration, the critic exactly evaluates the current actor. Together they define an exponential tilt, held fixed while completing sampling, distillation, and evaluation. With all three exact, the next actor equals the tilted policy and the next critic evaluates it, recovering ordinary PMD at every iteration.

With finite sampling, exact distillation and evaluation still align the next actor and critic to the sampler distribution, but the update only approximates PMD. Distillation or evaluation errors can break this consistency. A subsequent iteration can define a new guided distribution from the current actor and critic; accurate sampling, distillation, and evaluation can then restore consistency with that new policy. This does not guarantee recovery of the exact PMD trajectory.

Computational cost. Matching diffusion depth alone is insufficient. We report network-operation counts and full-loop timings with 100 levels, B=256B=256, one environment, and one training iteration per transition. Native rules are retained: MGMD uses one MALA update per level; DIPO uses 20 critic-gradient refinement steps; DPMD samples 32 candidates per state and updates its actor every other iteration. Each critic operation includes both QQ networks.

Operation per iterationMGMDDIPODPMD
Policy forward202B202B101B+100101B+1003200(B+1)+B/23200(B+1)+B/2
Policy input gradient202B202B0000
Policy parameter gradient00BBB/2B/2
Policy Hessian-vector product201B201B0000
Policy mixed-derivative productBB0000
Critic forward204B204B22B22B35B+3235B+32
Critic input gradient201B201B20B20B00
Critic parameter gradientBBBBBB

MGMD counts include guided energy and drift evaluations at both current and proposed actions. Acceptance arithmetic reuses these evaluations. Energy values and input gradients share forwards; second-order products use automatic differentiation without constructing Hessians and are counted separately.

Timings use one run on one H100 and average 250 synchronized iterations after compilation and warmup. They include action generation, environment interaction/resets, replay, transfers, and learning, excluding compilation, warmup, separate evaluation, external logging, and checkpointing. These are native-implementation timings (JAX for MGMD/DPMD; PyTorch for DIPO).

Time (ms/iteration)MGMDDIPODPMD
HalfCheetah30.89102.1313.45
Humanoid37.34100.5317.75

CGPO. CGPO shares the KL-regularized motivation and distills guided actions, but its normalized-gradient DSG rule does not generally realize the PMD tilt, and its guidance strength has no general calibration to the tilt coefficient. MGMD specifies intermediate target densities and uses Metropolis kernels that preserve them at fixed step sizes. CGPO also changes the critic and value-calibration machinery, so an end-to-end comparison would not isolate policy improvement. DPMD is our primary objective-matched baseline because it targets the same PMD update with online diffusion policies.

Langevin ablation. Across six fixed step sizes and ten seeds, MGMD outperforms every tested unadjusted Langevin setting on four of six tasks; selected settings outperform it on the other two. This supports greater robustness across the tested tasks. The comparison changes both Metropolis correction and step-size adaptation, so it does not isolate acceptance correction alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.