- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies policies defined by terminal laws of state-conditioned SDEs with drift Lipschitz budget . Under nondegenerate noise, it establishes matching squared-localization bounds and related value guarantees. For ReLU drifts of size , an oracle inequality for empirical return maximization over shared simulator randomness combines , , and estimation terms . This yields a rate of , improved to under dissipativity, where is the state dimension. A Girsanov-based policy-gradient identity and two synthetic experiments illustrate the generic and dissipative regimes.
Critical strengths
The paper gives matching localization bounds for a precisely specified SDE policy class. Its Girsanov-based experiments use a consistent policy parameterization, providing a coherent study of this restricted setting.
Critical weaknesses
-
The statistical result supports a narrower setting than the claimed RL learning trade-off. Theorem 4.1 provides an oracle guarantee for policy search with a replayable transition simulator, shared random scenarios, and global empirical return maximization. This is a meaningful planning result. Its applicability to the experimental policy-gradient updates, total interaction budget, and cited diffusion RL methods remains unestablished. Moreover, the estimation upper bound alone does not establish an intrinsic statistical cost of greater expressivity. The closely related PEGASUS framework (Ng and Jordan, 2000) is also not discussed or cited.
-
The expressivity analysis covers a restricted SDE policy class. The localization results address this class, and directly learning its drift is consistent with the Girsanov-based experiments. The class imposes uniform finite- drift regularity and persistent nondegenerate noise, without requiring correspondence to a forward noising process. Policies implementing reverse denoising dynamics can have extremely large or unbounded drift Lipschitz constants near the data endpoint. The paper does not distinguish these settings or establish how its drift budget characterizes terminal policy-distribution complexity. This limits the support for its broader diffusion-policy interpretation.
Questions for authors
-
Would the authors explicitly restrict the statistical claims to simulator-based policy search, or establish their applicability to the cited RL learning methods?
-
Which cited methods fit the analyzed SDE class, and would the authors restrict the diffusion-policy interpretation accordingly? If broader claims are retained, how does characterize terminal distribution complexity for policies tied to a forward noising process?
3 comments
Rebuttal for Reviewer yTWV (continued)
3. Connection to diffusion policies and reverse-denoising dynamics.
Our framework includes reverse-denoising SDEs whenever their drift, noise, and initialization satisfy the stated assumptions. This connection is structural: in the standard reverse-SDE formulation, the learned score enters the drift, while the noise schedule determines the diffusion coefficient. Most importantly, our localization lower bound applies generally to admissible reverse-denoising dynamics regardless of whether the drift was learned through score matching, policy gradients, or another procedure. Song et al., Section 3.2.
Requiring correspondence to a particular forward process is unnecessary for this lower bound. The theorem holds over the larger class of admissible -Lipschitz drifts, so it also holds over any reverse-score subclass contained within that class. Thus, the general SDE formulation gives a limitation that applies across different constructions of the drift.
Exact reverse-denoising dynamics can have an unbounded Lipschitz constant near the final sampling time. Our analysis asks how accurately diffusion policies can approximate optimal actions when the drift has a finite Lipschitz budget. For the RL objective, reproducing an ideal terminal distribution exactly is unnecessary: our value bounds show that sufficiently accurate concentration near optimal actions already yields nearly optimal performance.
Randomized policies play an established role in RL: they support exploration and allow the learner to retain multiple promising actions rather than commit prematurely to a single choice. Entropy-regularized methods such as Soft Actor-Critic Haarnoja et al., 2018 explicitly encourage policy stochasticity alongside reward maximization. This motivation is general to RL and is not specific to diffusion policies. Our persistent-noise setting is motivated by this broader role of randomization. The theoretical question is how accurately a diffusion policy can concentrate near optimal actions while its generating dynamics remain subject to nondegenerate noise. Our results quantify this relationship through the drift Lipschitz budget . As a concrete example of maintaining sampling noise in diffusion-based RL, DPPO imposes a positive minimum noise level during training to encourage exploration. DPPO, Section 4.3.
Finally, directly measures the regularity of the generating dynamics and controls a concrete aspect of the terminal distribution: its attainable concentration near optimal actions under nondegenerate noise. The admissible class permits nonlinear drifts and multimodal terminal laws. Our finite-sample analysis then connects this concentration benefit to statistical estimation, making a key parameter for studying approximation and learning jointly. We will clarify this interpretation without presenting as a complete measure of terminal-distribution complexity.
follow-up
We hope that our responses and revisions have addressed your concerns. Please let us know if you have any further questions or comments before the discussion period ends; we would be happy to address them!
We believe the paper offers a new quantitative perspective on diffusion-policy learning by identifying the drift Lipschitz budget as a key parameter connecting approximation power and finite-sample learning guarantees. We are excited about the possibilities this perspective opens up, and hope the revised manuscript conveys its significance more clearly.
Rebuttal for Reviewer yTWV
We thank the reviewer for recognizing the matching localization bounds and the coherence of the SDE policy parameterization. We believe two distinctions clarify the scope and significance of our results: the distinction between a policy class and its training algorithm, and between a finite drift budget and an exact singular-endpoint limit.
1. Statistical guarantee, simulator access, and PEGASUS.
We thank the reviewer for highlighting the common-randomness construction and the connection to PEGASUS.. We emphasize that shared randomness is a coupling tool used in our proof/analysis. The coupling itself does not restrict the policy class or change the trajectory distribution induced by any individual policy.
Specifically, let Gf(ξ) denote the return obtained by applying policy f to exogenous randomness ξ. For every fixed f, the resulting trajectory has exactly the distribution prescribed by the original MDP and policy, so E[Gf(ξ)]=V(f). Using independent scenarios ξ1,…,ξn, we define Vn(f)=n1∑i=1nGf(ξi).
Sharing these scenarios across policies specifies their joint empirical process while preserving each policy’s marginal trajectory law. This plays the same role as evaluating different candidate models on a common dataset in standard statistical ERM.
Our goal is to characterize the regret behavior of the optimal finite-sample policy. The nuance that makes the setup nontrivial is precisely that different policies will generate different distributions of actions and trajectories under the shared scenarios. Therefore, to compare between policies in a canonical way, we have to couple using the environment noise mechanism, which does not change with the policies.
We will clarify this role of shared randomness and discuss PEGASUS as a relevant antecedent. Our contribution is the quantitative approximation–estimation analysis of diffusion-policy ERM under this explicit, common-data formulation.
The reviewer raises a question about the scope of our SDE policy class. Our main result is that for any prescribed positive performance tolerance, this class contains a policy that attains that tolerance with a finite drift Lipschitz budget, under the stated assumptions. The distinction lies in the objective. In diffusion-based distribution recovery, the goal is to reproduce a prescribed target law, and the exact reverse drift can have a diverging Lipschitz constant near the data endpoint when that law is singular. In RL, a diffusion policy is evaluated by its expected return. Lemma 3.1 constructs a policy with drift Lipschitz constant at most K throughout the sampling interval and mean-squared localization error O(1/K) around an optimal action. Our value bounds show why this concentration suffices for near-optimal performance.
Regarding the reviewer’s concern about “how its drift budget characterizes terminal policy-distribution complexity,” our results characterize how tightly the terminal action law can concentrate under persistent noise. Proposition 3.3 gives a lower bound of order 1/K for localization near a fixed finite set of actions, matching the order achieved by our construction. This lower bound holds for every admissible drift, including any reverse-denoising drift satisfying the hypotheses, so it requires no correspondence to a prescribed forward noising process. We will clarify that our sharp characterization concerns localization near optimal actions, the property that enters our RL value bounds.