- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Generalized Optimal Policy Matching (GOPM), which formulates policy optimization as divergence minimization toward a soft-optimal policy, and develops Diffusion Advantage Matching (DAM) as its diffusion-policy instantiation. DAM uses a trajectory-level log-ratio objective with detached sampling and a leave-one-out advantage estimator to avoid backpropagation through the denoising chain. The method is further extended to offline and hybrid learning through a log-variance formulation and different sampling and critic-estimation strategies. Experiments evaluate online continuous control, offline D4RL tasks, hybrid learning, and a case study on distributional coverage on a Gaussian-mixture.
Critical strengths
-
A well-motivated and useful approach to diffusion-policy optimization. The detached trajectory-level objective provides a practical way to update diffusion policies without backpropagating through the entire denoising process. Its integration into online actor–critic learning is appealing and has the potential to improve computational efficiency and training stability.
-
Promising online performance on challenging tasks. The reported results show competitive performance against existing diffusion-policy methods, with encouraging improvements on challenging, high-dimensional control tasks. These findings suggest that the proposed approach has practical value for online reinforcement learning.
-
Evaluation beyond average control return. The paper examines distributional coverage alongside policy performance and evaluates the method across online, offline, and hybrid learning settings. The coverage analysis and policy-parameterization ablation provide useful perspectives on the method’s behavior and the benefits of expressive diffusion policies.
Critical weaknesses
-
The methodological contribution relative to VarGrad and RTB is insufficiently established. The central ingredients of the detached estimator—log-variance optimization, stopped sampling gradients, leave-one-out control variates, and the connection to reverse-KL gradients—are closely related to [1]. More directly, [2] introduce Relative Trajectory Balance (RTB), including trajectory-level log-ratio optimization without differentiating through sampling. The online extension may be useful, but the main text does not clearly isolate the additional algorithmic contribution. RTB is also absent from the reported offline and distribution-matching comparisons.
-
The transition from trajectory-level optimization to action-level optimality requires additional justification. Section 4 uses the DPI to relate trajectory and action KL divergences, then asserts recovery of the original soft-optimal action policy. However, the trajectory KL contains an additional conditional-trajectory KL term. An upper bound alone therefore does not establish equivalence of the optima; appropriate tightness and realizability conditions are needed. It is unclear from the main text whether these conditions hold for the finite-step diffusion parameterization with a fixed initial noise distribution.
-
The numerical experiments do not sufficiently isolate the source of the gains. Section 5.1.1 attributes the improvement over DIME to the estimator, citing the same architecture and reverse-KL objective. A controlled comparison within the same implementation, changing only the gradient estimator, would provide stronger evidence. This is especially relevant given the overlap with existing VarGrad/LOO estimators and the trajectory-based formulation of [3]. Matched environment interactions also do not establish computational efficiency such as runtime, peak memory, and the cost of multiple trajectories per state should be reported.
References
[1] Lorenz Richter et al. VarGrad: A Low-Variance Gradient Estimator for Variational Inference. NeurIPS, 2020.
[2] Siddarth Venkatraman et al. Amortizing Intractable Inference in Diffusion Models for Vision, Language, and Control. NeurIPS, 2024.
[3] Onur Celik, et al. DIME: Diffusion-Based Maximum Entropy Reinforcement Learning. ICML, 2025.
Questions for authors
-
Could the authors provide a more detailed qualitative comparison with RTB’s offline VarGrad formulation? Which differences beyond the online critic updates are necessary for the reported improvements, and can their effects be isolated experimentally?
-
Could the authors provide argument require the conditional-trajectory KL gap to vanish, and is this attainable under the implemented diffusion parameterization? Please also clarify whether the Section 4.1 critic estimates the action-regularized or trajectory-regularized value function.
-
What exactly is invariant to the sampling distribution? Is the intended claim limited to an attainable zero-loss solution under sufficient support? How does the support-based explanation of the offline results account for the policy component in the mixture sampler and for errors in the IQL critic outside the dataset?
I would be willing to raise my score if the authors sufficiently address these concerns through clear qualitative justification and supporting empirical evidence.
5 comments
Part 2/3 — Trajectory KL and critic interpretation
Problem 2: From trajectory-level optimization to action-level optimality
[Weakness 2]: [...] Section 4 uses the DPI to relate trajectory and action KL divergences, then asserts recovery of the original soft-optimal action policy. However, the trajectory KL contains an additional conditional-trajectory KL term. An upper bound alone therefore does not establish equivalence of the optima; appropriate tightness and realizability conditions are needed. It is unclear from the main text whether these conditions hold for the finite-step diffusion parameterization with a fixed initial noise distribution.
[Question 2]: Could the authors provide argument require the conditional-trajectory KL gap to vanish, and is this attainable under the implemented diffusion parameterization? Please also clarify whether the Section 4.1 critic estimates the action-regularized or trajectory-regularized value function.
[Answer]: Optimizing a trajectory-level variational bound instead of the clean-action objective is standard: diffusion models are themselves trained by minimizing such a bound, which is not tight in general, and they still recover the data distribution. Trajectory-level bounds are likewise standard in diffusion RL, e.g. DIME [3].
The appendix already states when this bound is tight (App. A.6). Proposition 2 (Eq. 34) separates the trajectory KL into two KL terms: the clean-action KL, , and the conditional-trajectory gap, . Together with Eq. 34, Corollary 1 then states that the bound is tight exactly when the gap vanishes, and that driving the trajectory KL to zero enforces the clean-action optimum. For readability, we move this result into the main text.
Both KL terms in Eq. 34 can be driven to zero simultaneously when the reference's noising path forgets the clean action by its final step, as a diffusion forward process does (Sec. 2.1). Any clean-action distribution can then be paired with this path while keeping the fixed Gaussian start of Eq. 16, so the two terms decouple and can be minimized independently. Both terms then reach zero, the gap vanishes, and Corollary 1 gives .
This is attainable under the implemented parameterization. Our sampler follows DIME's (App. G.2): the policy starts from a fixed Gaussian prior, and DIME designs its noising process to end in that same Gaussian (DIME [3], Sec. 3.2). We follow this design, so the reference ends where the policy starts. This holds for any variance of the Gaussian, so the choice of variance in Eq. 16 imposes no restriction. Moreover, Theorem 2 (App. A.4) applies with the whole chain as the action, so every DAM update is a soft policy improvement step for the trajectory-regularized objective, which lower-bounds the action-level one, as is standard for diffusion models.
The critic. The Sec. 4.1 critic estimates the trajectory-regularized value: its target (Eq. 20) penalizes the trajectory log-ratio, consistent with Theorems 1–2 applied with the whole chain as the action.
[3] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning, Celik et al., ICML 2025.
Part 3/3 — Estimator, computational cost, and sampling
Problem 3: Isolating the source of the gains, and computational cost
[Weakness 3]: The numerical experiments do not sufficiently isolate the source of the gains. Section 5.1.1 attributes the improvement over DIME to the estimator, citing the same architecture and reverse-KL objective. A controlled comparison within the same implementation, changing only the gradient estimator, would provide stronger evidence. This is especially relevant given the overlap with existing VarGrad/LOO estimators and the trajectory-based formulation of [3]. Matched environment interactions also do not establish computational efficiency such as runtime, peak memory, and the cost of multiple trajectories per state should be reported.
[Answer]: The comparison with DIME is controlled (App. E.3, l.1555–1560): both share the implementation, architecture and training protocol (App. G.2, G.6.1; Table 7) and the same trajectory-level RL objective (Eq. 3; [3]), and DAM's actor gradients are two orders of magnitude smaller (Table 5) while its returns are higher (Fig. 2). To isolate the estimator, as suggested, we keep the DAM loss fixed and change only the gradient estimator: the detached estimator reaches 11849 684 on Humanoid, whereas backpropagating through the sampler fails on all five seeds (387 52).
Computational cost is already reported in App. E.3: DAM is faster than five of the seven baselines (Table 6b), and even with two trajectories per state it outperforms DIME with less wall-clock time (Table 6a). We add peak memory for each number of trajectories per state to App. E.3.
[3] DIME: Diffusion-Based Maximum Entropy Reinforcement Learning, Celik et al., ICML 2025.
Problem 4: What is invariant to the sampler, and the support explanation
[Question 3]: What exactly is invariant to the sampling distribution? Is the intended claim limited to an attainable zero-loss solution under sufficient support? How does the support-based explanation of the offline results account for the policy component in the mixture sampler and for errors in the IQL critic outside the dataset?
[Answer]: Yes, the claim is limited to the attainable zero-loss solution. “Independent of the sampling distribution” (Intro, l.065–067; Conclusion, l.484–485) is a slip in wording on our part. What is invariant is the zero-loss solution, which every sampling distribution covering the policy and the reference shares, provided it is attainable (Problem 2). What we meant is flexibility: the sampler can draw on the current policy, the offline data, or both. We revise both sentences accordingly.
We would like to clarify that offline learning is an additional capability supported by DAM, rather than its primary contribution, which is a stable method for online KL-regularized RL that supports both Gaussian and diffusion policies trained from scratch, with guaranteed soft policy improvement (Theorems 1–2).
The policy component adds samples, not support. The target is the reference tilted by the critic, so it has the reference's support, which offline is a behaviour clone of the data (App. B.1, l.1153–1156). The policy component can move beyond the recorded actions, but only within this support. Its role is to supply the policy's own trajectories, so that the critic's ranking enters every update (Problem 1). The coverage that matters is therefore the dataset's, which is broadest for Medium-Replay by the D4RL design (App. G.1, l.1746–1749).
Errors in the IQL critic. Relying on an approximate IQL-type critic is standard in critic-guided offline methods, e.g. IDQL [4] and QGPO [5], and works well in practice, as it does for DAM (Table 2a); the paper treats it as a surrogate (Sec. 4.2, l.322–323). DAM's guarantee also does not need an exact critic: an offset shifts every residual of a state equally, leaving the loss and its minimizer unchanged (App. B.1, Eq. 36–37). As for errors outside the dataset, because the target stays within the behaviour-cloned reference's support, they matter only where that reference places mass, and a larger reduces their effect.
[4] IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies, Hansen-Estruch et al., 2023.
[5] Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline RL, Lu et al., ICML 2023.
Part 1/3 — RTB and VarGrad comparison
We thank the reviewer for the detailed review and for the openness to raising the score. We particularly appreciate the recognition that DAM's “integration into online actor–critic learning is appealing and has the potential to improve computational efficiency and training stability”, with “encouraging improvements on challenging, high-dimensional control tasks”. DAM differs from VarGrad and RTB well beyond this online integration, in the problem it solves and its general design, as we make explicit below.
Problem 1: Contribution relative to VarGrad and RTB
[Weakness 1]: The methodological contribution relative to VarGrad and RTB is insufficiently established. [...] The online extension may be useful, but the main text does not clearly isolate the additional algorithmic contribution. RTB is also absent from the reported offline and distribution-matching comparisons.
[Question 1]: Could the authors provide a more detailed qualitative comparison with RTB's offline VarGrad formulation? Which differences beyond the online critic updates are necessary for the reported improvements, and can their effects be isolated experimentally?
[Answer]: We would like to clarify that DAM is not RTB with online critic updates: the two methods solve different problems. RTB [2] is limited to fine-tuning a pretrained diffusion model toward a fixed target via a predefined reward (RTB, Eq. 13). Its MDP is therefore not a general environment MDP but the denoising process of the diffusion model, and RTB is in this sense a variant of stochastic optimal control whose base process is a diffusion process, not an online RL method. VarGrad [1] is a variational-inference estimator toward a fixed, known density.
In contrast, DAM is designed for general RL: it learns a policy from scratch in an environment MDP, with a jointly learned critic, toward a target that moves with the policy (Eq. 3; Theorems 1–2). Our method applies to any policy class, with diffusion policies as one instance alongside Gaussian policies (App. E.1), whereas RTB is limited to diffusion models. As a result, a single DAM objective covers online learning (Table 1), offline learning (Table 2a), single-stage hybrid learning (Fig. 3; Sec. 5.1.3) and two-stage offline-to-online learning (App. E.2, Table 4), whereas RTB supports neither online nor hybrid learning. The distribution-matching comparison (Sec. 5.1.4) includes only methods that learn from scratch.
Offline DAM vs. RTB offline. Offline learning is a capability that DAM supports rather than its main contribution, and even here the two methods are designed differently. RTB fine-tunes on its own samples and makes little use of the data: its dataset trajectories carry no critic signal (App. B.1, Eq. 39), and adding them changes its scores by at most 1.2 points (RTB, Table G.1). DAM, in contrast, is built to learn from the data. Its objective is a divergence that remains valid under covering sampler (App. F), so its offline sampler is constructed around the dataset (Eq. 40–42) where the critic's ranking enters every update.
Empirically, trained on RTB-style dataset trajectories alone, the policy collapses on Hopper-Medium (17.8, vs. 89.2 for DAM). DAM also needs critic values only, whereas RTB additionally relies on the critic's action gradient ∇aQ (RTB, Eq. 11). With RTB added to Table 2a (RTB from its Table 4), DAM is ahead on 8 of the 9 datasets:
[1] VarGrad: A Low-Variance Gradient Estimator for Variational Inference, Richter et al., NeurIPS 2020.
[2] Amortizing Intractable Inference in Diffusion Models for Vision, Language, and Control, Venkatraman et al., NeurIPS 2024.