- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SNARL, a maximum-entropy RL framework that decouples action generation from policy log-density estimation. It trains a separate energy model using spatial and temporal score matching with Gaussian anchoring to approximate the policy's negative log-density, enabling SAC-style optimization with a one-step pushforward actor. Experiments demonstrate competitive control performance and substantially reduced action inference latency compared with diffusion-based policies.
Critical strengths
-
The paper addresses a meaningful challenge in maximum-entropy RL with implicit generative policies. The proposed Gaussian anchoring provides a reasonable approach to online normalization of state-conditioned energy models.
-
The experiments cover diverse continuous-control benchmarks and demonstrate competitive performance, particularly on humanoid locomotion tasks, with significantly reduced action inference latency.
Critical weaknesses
-
The core density estimation technique builds heavily on existing Dual Score Matching, with the main contribution being its adaptation to online RL through Gaussian anchoring. Moreover, the paper lacks comparisons with standard Gaussian SAC or simpler entropy-regularization alternatives using one-step actors. It remains unclear whether the additional energy modeling complexity is necessary to achieve the reported performance in the experiments.
-
One weakness is the insufficient validation of energy estimation accuracy on the actual policy. Although the paper evaluates energy estimation on synthetic distributions and a 29-dimensional Gaussian mixture policy, these experiments do not directly validate the accuracy of the learned energy model for the high-dimensional implicit actors used in the main RL benchmarks. Since accurate log-density estimation is central to the proposed method, stronger validation under realistic policy learning dynamics is needed.
-
The strongest improvements appear on MuJoCo Playground, where SNARL uses substantially different training configurations from the baselines, particularly a much larger replay buffer. These differences make it difficult to isolate the contribution of the proposed method. Furthermore, the reported efficiency gains focus on action inference latency, without adequately accounting for the additional computational cost of energy model training.
Questions for authors
-
How does SNARL compare with standard SAC and simpler entropy-regularized one-step actors under matched training settings?
-
Can the authors provide more direct evidence of energy estimation accuracy for the actual high-dimensional implicit policies during RL training?
-
Can the authors provide comparisons under more closely matched replay-buffer and computational budgets, including training wall-clock time?
2 comments
Dear Reviewer RvDT,
Thank you again for your review. As the discussion period ends today, we wanted to briefly check whether our response has addressed your concerns. In summary:
- Matched replay buffer (W3/Q3): we reran DIME and CrossQ with matched replay buffer size, batch size, and interaction budget. SNARL still achieves the highest return on all four Playground tasks (Experiment 1).
- Gaussian SAC baseline (W1/Q1): we added CrossQ on all three benchmarks (Experiments 1 and 2).
- Energy accuracy on the actual policy (W2/Q2): on the 23- and 29-dimensional G1 and T1 tasks, settles within about 1 nat of zero during training (Experiment 3).
- Training cost (W3/Q3): energy learning adds 22% training time, while action inference is 4–14× faster than the diffusion- and flow-based baselines.
If any concern remains, we would be glad to clarify before the discussion closes. If our response has resolved them, we would be grateful if you would consider updating your assessment.
Note. All new experiments referenced below (Experiments 1–5, with figures and tables) are in the Additional Experiments PDF uploaded as supplementary material.
W1 and Q1. Relation to DSM. Thank you for raising this point. We agree that the estimator builds on DSM. Our contribution is what is needed to use it in online RL. DSM estimates a fixed log-density up to a constant shift, and only needs to estimate that constant once after training. A policy instead requires a state-dependent offset c(s) to be controlled during training, because the energy enters the soft Bellman target while the policy is still changing. The anchor loss is what provides this: without it, returns on dog-trot fall below even the no-entropy variant (Figure 5a of the paper).
Gaussian SAC and simpler alternatives. We added CrossQ, a strong tanh-Gaussian SAC baseline, on all three benchmarks (Experiments 1 and 2 in the supplementary material). It is close to SNARL on simpler tasks such as HalfCheetah, but behind on all three DMC humanoid tasks and all four Playground tasks. The gap is largest on the two T1 tasks: CrossQ does not learn, whereas both the diffusion baseline DIME and SNARL do. This is consistent with expressive policies being beneficial on these tasks, although the comparison is not a controlled test of the policy class alone. A one-step actor without the energy model also performs worse (Figure 5a of the paper). Together these results support the effectiveness of learned entropy regularization for our expressive one-step actor.
W2 and Q2. We agree that the Gaussian-mixture studies in the paper do not directly validate the energy for the implicit actors used in RL. To evaluate the actual implicit policy, we ran an additional experiment on the G1 and T1 Joystick tasks, which have 29 and 23 action dimensions (5 seeds per task). We use a kernel density estimate as the reference: an RBF kernel with bandwidth h=0.01 and 65,536 sampled actions per state. We report the relative error between the energy prediction and this reference, along with nested importance-sampling estimates of the log-normalizer logZU(s).
Across the four tasks, logZU(s) settles within about 1 nat of zero during training, and the relative fit error stabilizes after an early decrease on the G1 tasks. This is the same fit metric as in the Gaussian-mixture experiments of the paper (Appendix D.3). The curves are in Figure 4 (Experiment 3) in the supplementary material.
W3 and Q3. Replay buffer. We agree that results reported in prior work, here with smaller replay buffers, may not be as comparable as ones we reproduce. FPO, REPPO, and AMDP are on-policy algorithms, and their small replay buffers follow the settings reported in their respective publications. DIME and CrossQ are off-policy algorithms, like SNARL, so we reran both with matched configurations, including replay buffer size, batch size, and interaction budget. The table below reports final IQM returns with 95% bootstrap confidence intervals. DIME (Reported) is from the AMDP study, and the matched runs are newly added with 5 seeds each.
The matched configuration raises DIME's IQM on all four tasks and closes part of the gap shown in the paper, while SNARL still achieves the highest return on all four.
Training cost. We agree that SNARL introduces additional training cost. The table below reports the average training time for 1M steps on the six MuJoCo tasks, measured on the same NVIDIA A6000 GPU. Inference latency is as reported in Table 1 of the paper. "—" denotes not measured.
Energy learning, including double backpropagation, adds 22% training time. SNARL trains slower than MFPO, comparably to SAC Flow, and faster than FlowRL and DIME. The efficiency advantage we claim is therefore in deployment: SNARL outperforms the diffusion- and flow-based baselines in return on some tasks and matches them on others, but is always much faster to deploy, with 4–14× lower action-inference latency.