acceptodds
Under review as a conference paper at ICLR 2027

SNARL: Self-Normalizing Anchored Maximum-Entropy Reinforcement Learning

19 views

Abstract

Expressive generative policies can capture complex action distributions, but maximum-entropy (MaxEnt) reinforcement learning requires evaluating the policy log-density , which is generally intractable for implicit generators. We introduce Self-Normalizing Anchored MaxEnt Reinforcement Learning (SNARL), an off-policy framework that learns a separate state-conditioned energy model to estimate the policy's negative log-density from samples. SNARL augments spatial and temporal score matching with an anchor derived from the Gaussian limit of variance-preserving corruption, approximately normalizing the energy model as the policy evolves. The resulting model estimates the policy's negative log-density in a single forward pass. By separating policy log-density estimation from action generation, SNARL enables maximum-entropy learning with a one-step pushforward actor that generates each action in a single network evaluation. Experiments show that SNARL is competitive with strong diffusion and flow baselines across high-dimensional continuous-control benchmarks, excels on MuJoCo Playground humanoid locomotion tasks, and provides approximately faster policy inference than DIME.

Primary area:
reinforcement learning
Keywords:
Maximum-Entropy Reinforcement Learning, Energy-Based Models, Implicit Policies, Policy Density Estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RvDTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SNARL, a maximum-entropy RL framework that decouples action generation from policy log-density estimation. It trains a separate energy model using spatial and temporal score matching with Gaussian anchoring to approximate the policy's negative log-density, enabling SAC-style optimization with a one-step pushforward actor. Experiments demonstrate competitive control performance and substantially reduced action inference latency compared with diffusion-based policies.

Critical strengths
  1. The paper addresses a meaningful challenge in maximum-entropy RL with implicit generative policies. The proposed Gaussian anchoring provides a reasonable approach to online normalization of state-conditioned energy models.

  2. The experiments cover diverse continuous-control benchmarks and demonstrate competitive performance, particularly on humanoid locomotion tasks, with significantly reduced action inference latency.

Critical weaknesses
  1. The core density estimation technique builds heavily on existing Dual Score Matching, with the main contribution being its adaptation to online RL through Gaussian anchoring. Moreover, the paper lacks comparisons with standard Gaussian SAC or simpler entropy-regularization alternatives using one-step actors. It remains unclear whether the additional energy modeling complexity is necessary to achieve the reported performance in the experiments.

  2. One weakness is the insufficient validation of energy estimation accuracy on the actual policy. Although the paper evaluates energy estimation on synthetic distributions and a 29-dimensional Gaussian mixture policy, these experiments do not directly validate the accuracy of the learned energy model for the high-dimensional implicit actors used in the main RL benchmarks. Since accurate log-density estimation is central to the proposed method, stronger validation under realistic policy learning dynamics is needed.

  3. The strongest improvements appear on MuJoCo Playground, where SNARL uses substantially different training configurations from the baselines, particularly a much larger replay buffer. These differences make it difficult to isolate the contribution of the proposed method. Furthermore, the reported efficiency gains focus on action inference latency, without adequately accounting for the additional computational cost of energy model training.

Questions for authors
  1. How does SNARL compare with standard SAC and simpler entropy-regularized one-step actors under matched training settings?

  2. Can the authors provide more direct evidence of energy estimation accuracy for the actual high-dimensional implicit policies during RL training?

  3. Can the authors provide comparisons under more closely matched replay-buffer and computational budgets, including training wall-clock time?

2 comments

Note. All new experiments referenced below (Experiments 1–5, with figures and tables) are in the Additional Experiments PDF uploaded as supplementary material.

W1 and Q1. Relation to DSM. Thank you for raising this point. We agree that the estimator builds on DSM. Our contribution is what is needed to use it in online RL. DSM estimates a fixed log-density up to a constant shift, and only needs to estimate that constant once after training. A policy instead requires a state-dependent offset c(s)c(s) to be controlled during training, because the energy enters the soft Bellman target while the policy is still changing. The anchor loss is what provides this: without it, returns on dog-trot fall below even the no-entropy variant (Figure 5a of the paper).

Gaussian SAC and simpler alternatives. We added CrossQ, a strong tanh-Gaussian SAC baseline, on all three benchmarks (Experiments 1 and 2 in the supplementary material). It is close to SNARL on simpler tasks such as HalfCheetah, but behind on all three DMC humanoid tasks and all four Playground tasks. The gap is largest on the two T1 tasks: CrossQ does not learn, whereas both the diffusion baseline DIME and SNARL do. This is consistent with expressive policies being beneficial on these tasks, although the comparison is not a controlled test of the policy class alone. A one-step actor without the energy model also performs worse (Figure 5a of the paper). Together these results support the effectiveness of learned entropy regularization for our expressive one-step actor.

W2 and Q2. We agree that the Gaussian-mixture studies in the paper do not directly validate the energy for the implicit actors used in RL. To evaluate the actual implicit policy, we ran an additional experiment on the G1 and T1 Joystick tasks, which have 29 and 23 action dimensions (5 seeds per task). We use a kernel density estimate as the reference: an RBF kernel with bandwidth h=0.01h = 0.01 and 65,536 sampled actions per state. We report the relative error between the energy prediction and this reference, along with nested importance-sampling estimates of the log-normalizer log⁡ZU(s)\log Z_U(s).

Across the four tasks, log⁡ZU(s)\log Z_U(s) settles within about 1 nat of zero during training, and the relative fit error stabilizes after an early decrease on the G1 tasks. This is the same fit metric as in the Gaussian-mixture experiments of the paper (Appendix D.3). The curves are in Figure 4 (Experiment 3) in the supplementary material.

W3 and Q3. Replay buffer. We agree that results reported in prior work, here with smaller replay buffers, may not be as comparable as ones we reproduce. FPO, REPPO, and AMDP are on-policy algorithms, and their small replay buffers follow the settings reported in their respective publications. DIME and CrossQ are off-policy algorithms, like SNARL, so we reran both with matched configurations, including replay buffer size, batch size, and interaction budget. The table below reports final IQM returns with 95% bootstrap confidence intervals. DIME (Reported) is from the AMDP study, and the matched runs are newly added with 5 seeds each.

TaskDIME (Reported)DIME (Matched)CrossQ (Matched)SNARL
G1 Flat9.48 [5.76, 11.71]22.90 [20.95, 24.61]20.60 [17.47, 24.29]39.16 [38.12, 39.58]
G1 Rough7.54 [4.07, 10.79]16.02 [14.37, 18.54]15.86 [6.67, 19.00]33.49 [32.48, 34.48]
T1 Flat24.52 [19.94, 29.42]26.54 [19.81, 28.95]0.01 [0.00, 5.29]39.68 [39.45, 40.07]
T1 Rough15.40 [11.33, 20.17]24.17 [21.46, 26.17]0.01 [0.00, 0.02]36.00 [34.88, 36.49]

The matched configuration raises DIME's IQM on all four tasks and closes part of the gap shown in the paper, while SNARL still achieves the highest return on all four.

Training cost. We agree that SNARL introduces additional training cost. The table below reports the average training time for 1M steps on the six MuJoCo tasks, measured on the same NVIDIA A6000 GPU. Inference latency is as reported in Table 1 of the paper. "—" denotes not measured.

MethodTraining time (min)Training time relative to SNARLInference latency (ms)
MFPO940.53×0.148
SNARL w/o Energy1440.82×—
SNARL1761.00×0.036
SAC Flow1841.05×0.190
FlowRL1951.11×0.332
DIME2281.30×0.500

Energy learning, including double backpropagation, adds 22% training time. SNARL trains slower than MFPO, comparably to SAC Flow, and faster than FlowRL and DIME. The efficiency advantage we claim is therefore in deployment: SNARL outperforms the diffusion- and flow-based baselines in return on some tasks and matches them on others, but is always much faster to deploy, with 4–14× lower action-inference latency.

Dear Reviewer RvDT,

Thank you again for your review. As the discussion period ends today, we wanted to briefly check whether our response has addressed your concerns. In summary:

  • Matched replay buffer (W3/Q3): we reran DIME and CrossQ with matched replay buffer size, batch size, and interaction budget. SNARL still achieves the highest return on all four Playground tasks (Experiment 1).
  • Gaussian SAC baseline (W1/Q1): we added CrossQ on all three benchmarks (Experiments 1 and 2).
  • Energy accuracy on the actual policy (W2/Q2): on the 23- and 29-dimensional G1 and T1 tasks, log⁡ZU(s)\log Z_U(s) settles within about 1 nat of zero during training (Experiment 3).
  • Training cost (W3/Q3): energy learning adds 22% training time, while action inference is 4–14× faster than the diffusion- and flow-based baselines.

If any concern remains, we would be glad to clarify before the discussion closes. If our response has resolved them, we would be grateful if you would consider updating your assessment.

Reviewer_wb2pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SNARL, an off-policy maximum-entropy reinforcement learning framework that estimates policy log-densities using a separate state-conditioned energy model. By combining spatial and temporal score matching with a Gaussian endpoint anchor under variance-preserving corruption, SNARL aims to maintain approximate energy normalisation as the policy evolves. This decouples density estimation from action generation and enables a one-step actor requiring only one network function evaluation (NFE) per action. The experiments validate the approach through controlled density-estimation and tracking experiments, continuous-control benchmarks, and ablation studies. SNARL achieves competitive returns on MuJoCo and DeepMind Control tasks and outperforms the evaluated baselines on all four MuJoCo Playground humanoid tasks. Its reported inference latency is approximately 14 times lower than that of DIME. These results are promising, although the extent of the advantage over existing methods warrants further scrutiny.

Critical strengths
  1. The Gaussian anchoring mechanism addresses the state-dependent additive ambiguity inherent in score-based energy estimation. Controlled experiments demonstrate a substantial improvement in normalisation, providing empirical support for the central methodological contribution.

  2. SNARL achieves competitive performance across diverse continuous-control benchmarks while generating actions in a single network evaluation. The reported latency reduction is practically relevant for real-time control, and the ablations provide evidence for the importance of both anchoring and sufficiently frequent energy updates.

  3. This paper is well organized and clear presented including detailed experiment settings.

Critical weaknesses
  1. Limited evidence for the reliability of density estimation in complex, high-dimensional settings. The direct normalisation evaluation primarily uses a two-dimensional Gaussian mixture, while the higher-dimensional tracking experiments remain controlled synthetic tests. Since the estimated energy directly affects the critic, actor and temperature updates, errors in its state-dependent normalisation could materially distort learning. Further evidence of estimation accuracy during realistic policy evolution would strengthen confidence in the method.

  2. Incomplete evidence for the comparative performance claims. Some benchmark baselines, particularly those on DeepMind Control and MuJoCo Playground, are taken from prior work rather than reproduced under a common experimental protocol. Differences in evaluation procedures and training configurations may therefore complicate the comparisons. Moreover, the inference-speed analysis does not quantify the additional training cost of the energy network, which uses double backpropagation and multiple updates per actor update. The overall computational advantage is consequently not fully established.

  3. Benchmark baselines miss some important method like QVPO, CrossQ.

Questions for authors

See Weakness above

3 comments

Note. All new experiments referenced below (Experiments 1–5, with figures and tables) are in the Additional Experiments PDF uploaded as supplementary material.

W1. Thank you for this question. Our controlled studies use Gaussian mixtures, including a 29-dimensional one (Appendix D.5), because they provide exact log-densities to compare against. To evaluate the actual implicit policy, we ran an additional experiment on the G1 and T1 Joystick tasks, which have 29 and 23 action dimensions (5 seeds per task). We use a kernel density estimate as the reference: an RBF kernel with bandwidth h=0.01h = 0.01 and 65,536 sampled actions per state. We report the relative error between the energy prediction and this reference, along with nested importance-sampling estimates of the log-normalizer log⁡ZU(s)\log Z_U(s).

Across the four tasks, log⁡ZU(s)\log Z_U(s) settles within about 1 nat of zero during training, and the relative fit error stabilizes after an early decrease on the G1 tasks. This is the same fit metric as in the Gaussian-mixture experiments of the paper (Appendix D.3). The curves are in Figure 4 (Experiment 3) in the supplementary material.

W2. Comparison fairness. We agree that results reported in prior work may not be as comparable as ones we reproduce, and we will state this explicitly in the paper. All methods share the interaction budget and the metric (IQM with bootstrap confidence intervals), but the DMC and Playground baselines were run under their authors' configurations, so part of a performance gap could in principle come from the configuration rather than the method. For MuJoCo, all baselines in the submitted paper were reproduced by us from the official code with 10 seeds.

On Playground, where the settings differ most, FPO, REPPO, and AMDP are on-policy algorithms, and their small replay buffers follow the settings reported in their respective publications. DIME and CrossQ are off-policy algorithms, like SNARL, so we reran both with matched configurations, including replay buffer size, batch size, and interaction budget (Experiment 1 in the supplementary material, 5 seeds each). The matched configuration raises DIME's IQM on all four tasks, and SNARL still achieves the highest return on all four. The table below reports final IQM returns with 95% bootstrap confidence intervals. DIME (Reported) is from the AMDP study, and the matched runs are newly added with 5 seeds each.

TaskDIME (Reported)DIME (Matched)CrossQ (Matched)SNARL
G1 Flat9.48 [5.76, 11.71]22.90 [20.95, 24.61]20.60 [17.47, 24.29]39.16 [38.12, 39.58]
G1 Rough7.54 [4.07, 10.79]16.02 [14.37, 18.54]15.86 [6.67, 19.00]33.49 [32.48, 34.48]
T1 Flat24.52 [19.94, 29.42]26.54 [19.81, 28.95]0.01 [0.00, 5.29]39.68 [39.45, 40.07]
T1 Rough15.40 [11.33, 20.17]24.17 [21.46, 26.17]0.01 [0.00, 0.02]36.00 [34.88, 36.49]

Computational cost. We agree that SNARL introduces additional training cost. The table below reports the average training time for 1M steps on the six MuJoCo tasks, measured on the same NVIDIA A6000 GPU. Inference latency is as reported in Table 1 of the paper. "—" denotes not measured.

MethodTraining time (min)Training time relative to SNARLInference latency (ms)
MFPO940.53×0.148
SNARL w/o Energy1440.82×—
SNARL1761.00×0.036
SAC Flow1841.05×0.190
FlowRL1951.11×0.332
DIME2281.30×0.500

Energy learning, including double backpropagation, adds 22% training time. SNARL trains slower than MFPO, comparably to SAC Flow, and faster than FlowRL and DIME. The efficiency advantage we claim is therefore in deployment: SNARL outperforms the diffusion- and flow-based baselines in return on some tasks and matches them on others, but is always much faster to deploy, with 4–14× lower action-inference latency.

W3. We agree that both are important baselines and have added them. We ran CrossQ and QVPO on the six MuJoCo-v4 tasks and CrossQ on the seven DeepMind Control tasks, all with a 1M-step budget and 5 seeds each. We evaluated QVPO on MuJoCo only: it takes about 5 hours per run and is already below CrossQ on five of the six MuJoCo tasks, so we spent the remaining rebuttal compute on CrossQ for DMC and Playground. Learning curves are in Experiments 1 and 2 in the supplementary material.

CrossQ is a strong baseline on the simpler tasks. On the harder tasks SNARL is ahead: it achieves higher returns than CrossQ on all three DMC humanoid tasks, and on Playground CrossQ does not learn on the two T1 tasks under the matched configuration.

I think these new experiment results should update into main paper. I improve my score from 2 to 3.

Thank you for raising your score and for the valuable suggestions! We will incorporate the new experiments into the revised paper.

Reviewer_g7VzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses how to extend maximum-entropy RL, which requires entropy or log-density evaluation, to policies for which such evaluation is intractable. To estimate the log-density, the paper uses Dual Score Matching (DSM) to learn an energy model corresponding to the policy. Since applying DSM alone leaves a state-dependent bias in the energy function, the authors propose an anchor loss to resolve it. They then present SNARL, an algorithm that sequentially updates the energy model, critic, actor, and temperature. In experiments on MuJoCo, DeepMind Control, and MuJoCo Playground, SNARL achieves performance competitive with or exceeding the baselines despite using a one-step (1-NFE) actor, along with fast inference time.

Critical strengths

S1. Important problem, natural solution.

MaxEnt RL is widely used, yet it cannot be applied directly when the policy log-density is intractable, which is an important practical limitation. The proposed solution is natural: it builds on recent energy-learning methods (DSM) and adapts them to the RL setting through the anchor loss, which keeps the state-conditioned energy approximately normalized.

S2. Strong performance with a lightweight actor.

Despite using a 1-NFE actor, SNARL matches or outperforms diffusion- and flow-based baselines across the benchmarks, while achieving roughly 14× faster action inference than DIME. This supports the paper’s central claim that decoupling log-density estimation from action generation allows MaxEnt learning with a cheap actor without sacrificing performance.

Critical weaknesses

W1. Limited novelty relative to AR-DAE [1], which is not cited.

Training a separate model to track the policy’s log-density in MaxEnt RL was already proposed in AR-DAE (Section 6.3, Appendix G.2), which the paper does not cite. Both methods learn an energy-like model from current-policy samples and use it in the actor and critic updates. The main differences appear to lie in the training objective for the energy-like models and in how the state-dependent normalization is handled.

W2. Limited reliability of the experimental comparisons.

The baselines do not appear to be compared under identical settings. This is understandable when their results are reproduced from existing, but it still limits how reliable the comparisons are. In addition, many learning curves seem to be truncated before convergence, so final performance remains unclear. A controlled experiment with a policy class that admits exact entropy (e.g., a tanh-Gaussian policy) would make the results more reliable. It should compare (i) ground-truth entropy, (ii) SNARL’s estimated entropy, and (iii) no entropy, on the control benchmarks.

[1] Lim, Jae Hyun, et al. "AR-DAE: towards unbiased neural entropy gradient estimation." International Conference on Machine Learning. PMLR, 2020.

Questions for authors

Q1 (Clarification). Could the authors clarify how SNARL differs from AR-DAE, and what its novelty is relative to that work?

Q2 (Clarification). Are there any results that could address the concerns raised in Weakness 2? I understand that rerunning all experiments may not be feasible within the rebuttal period, so results on a few representative tasks would already be helpful.

Q3 (Clarification). In Figure 2, why does SNARL outperform the baselines by a fairly large margin on HalfCheetah? To my knowledge, HalfCheetah is an easier benchmark than Ant, yet the methods appear to saturate on Ant but not on HalfCheetah.

Q4 (Suggestion for future work). It may be worth scheduling G_π, i.e., using more energy updates per actor update early in training, when the policy distribution is expected to change faster, and fewer later on. This could reduce the training cost of the energy model without degrading log-density tracking.

2 comments

Note. All new experiments referenced below (Experiments 1–5, with figures and tables) are in the Additional Experiments PDF uploaded as supplementary material.

W1 and Q1. Thank you for pointing out AR-DAE. We will add the citation and a discussion in the revised version. AR-DAE indeed combines an implicit policy with an auxiliary energy model for MaxEnt RL. Our novelty therefore does not lie in using an estimated energy model for MaxEnt RL, but in how we obtain a normalized energy estimate. The two methodological differences are: (1) Normalization. AR-DAE focuses on estimating the gradient of the log-density, so its model gives an unnormalized log-probability, and the normalization constant Z(s)Z(s) is estimated separately through importance sampling (Appendix G.2 of [1]). SNARL instead jointly trains VP space- and time-score matching with Gaussian anchoring, which calibrates the absolute level of the energy online. (2) Computation. As a result, SNARL can provide an approximate log-density estimate in a single forward pass of the energy network, avoiding the separate normalization step, which uses 10 or 100 additional samples in AR-DAE (Table 9 of [1]).

[1] Lim et al., AR-DAE: Towards Unbiased Neural Entropy Gradient Estimation, ICML 2020.

W2 and Q2. We agree that results reported in prior work may not be as comparable as ones we reproduce, and we will state this explicitly in the paper. On Playground, where the settings differ most, FPO, REPPO, and AMDP are on-policy algorithms, and their small replay buffers follow the settings reported in their respective publications. DIME and CrossQ are off-policy algorithms, like SNARL, so we additionally ran both with matched configurations, including replay buffer size, batch size, and interaction budget (Experiment 1 in the supplementary material, 5 seeds each). SNARL still achieves the highest return on all four tasks.

In addition, many learning curves seem to be truncated before convergence, so final performance remains unclear.

We evaluate under fixed interaction budgets because sample efficiency is a central goal of off-policy MaxEnt RL. For MuJoCo and DMC, we use the 1M-step setting adopted in prior work such as SAC Flow and DIME. For MuJoCo Playground, GPU-parallel simulation makes a 50M-step budget practical. "Final returns" refers to returns at the end of these budgets. We acknowledge that these comparisons do not establish asymptotic performance rankings and will clarify this in the text and captions.

A controlled experiment with a policy class that admits exact entropy (e.g., a tanh-Gaussian policy) would make the results more reliable.

We implemented a tanh-Gaussian actor based on CrossQ, a strong Gaussian SAC baseline, and ran the experiment on two representative tasks: MuJoCo Humanoid-v4 and G1 Joystick Rough Terrain. Each variant uses five seeds and the same energy-update schedule as in the paper. The table below reports the IQM over the last five evaluations. Learning curves are in Experiment 4 in the supplementary material.

TaskVariantLast-5 IQM95% bootstrap CI
Humanoid-v4Exact10104.4[9479.1, 10384.2]
Humanoid-v4Learned Energy9580.8[8774.1, 9790.8]
Humanoid-v4No Entropy8469.4[7917.1, 9587.4]
G1 RoughExact15.86[6.67, 19.00]
G1 RoughLearned Energy15.95[12.80, 16.73]
G1 RoughNo Entropy1.51[0.31, 2.37]

On G1 Joystick Rough Terrain, the learned energy matches the exact entropy, while removing the entropy term collapses learning. On Humanoid-v4, the IQM ordering from highest to lowest is Exact, Learned Energy, and No Entropy, but the confidence intervals overlap, so with five seeds this task does not separate the variants. In short, where entropy regularization clearly matters (G1), the learned energy recovers the benefit of the exact entropy.

Q3. Thank you for raising this point. Our intuition is that HalfCheetah can be solved effectively by a relatively simple policy, so the additional complexity of multi-step generative policies may not translate into higher returns within 1M steps. The newly added Gaussian baseline CrossQ also achieves strong returns on HalfCheetah, close to SNARL's (Figure 2 in the supplementary material).

Q4. We agree, and this is a direction we plan to pursue: adaptively selecting the energy update frequency. This is a natural choice, since the policy changes at very different speeds over the course of training. Energy learning currently adds 22% to training time (Experiment 5 in the supplementary material), so such a schedule could recover part of this cost. The efficiency advantage we claim is in deployment: SNARL outperforms the diffusion- and flow-based baselines in return on some tasks and matches them on others, but is always much faster to deploy, with 4–14× lower action-inference latency (Table 1 of the paper).

I appreciate the authors’ additional experiments and detailed response.

My concern in W2 and my question in Q2 have been well addressed. The experiments under identical settings show SNARL’s performance gain more reliably, and the controlled experiment shows that the energy estimation is reliable. I also find the reason for using fixed interaction budgets convincing. I would encourage the authors to include these results in the camera-ready version.

For W1/Q1, the response clearly explains the difference from AR-DAE: normalization and the resulting computational advantage. However, I still find the novelty limited. Also, although the authors said they would add the citation and discussion, the manuscript has not been updated yet.

Overall, the rebuttal resolved my concern about experimental reliability and clarified the difference from AR-DAE, but not enough to raise my positive score. If a finer score scale were available, my assessment would be around 3.5.

Minor: the citations for FPO and REPPO seem to be missing. Please add them in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.