acceptodds
Under review as a conference paper at ICLR 2027

Expressivity and Statistical Trade-offs in Diffusion Policy Learning

Abstract

Diffusion-based policies have recently emerged as powerful policy parameterizations for reinforcement learning, but their theoretical properties remain largely unexplored. In this paper, we study the expressivity and finite-sample behavior of diffusion policies generated by state-conditioned diffusion processes with parameterized drifts. We show that diffusion policies with -Lipschitz drifts approximate optimal policies with error of order , where measures the complexity of the policy class, and establish a matching lower bound showing that this rate is optimal. This expressivity gain comes at a statistical cost: a larger Lipschitz budget improves approximation but makes learning more challenging, leading to a finite-sample trade-off. When the drift is parameterized by neural networks, we decompose the policy error into three components: diffusion approximation error, neural network realization error, and statistical estimation error. Balancing these terms yields a performance gap of order , where is the sample size and is the state dimension. We also discuss how additional smoothness and low-dimensional structure of the optimal policy can further sharpen these rates. Numerical experiments illustrate this approximation–estimation trade-off and suggest a practical design principle: choose the Lipschitz budget based on the available sample size, and then parameterize only the corresponding state dependence, leading to more efficient, sample-aware diffusion policy design.

Primary area:
reinforcement learning
Keywords:
diffusion models, reinforcement learning, expressivity, statistical tradeoff, lipschitz budget

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yTWVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies policies defined by terminal laws of state-conditioned SDEs with drift Lipschitz budget KK. Under nondegenerate noise, it establishes matching 1/K1/K squared-localization bounds and related value guarantees. For ReLU drifts of size ss, an oracle inequality for empirical return maximization over shared simulator randomness combines K−1K^{-1}, s−2/ms^{-2/m}, and estimation terms sK/n+sK/n\sqrt{sK/n}+sK/n. This yields a rate of O~(n−2/(m+6))\widetilde O(n^{-2/(m+6)}), improved to O~(n−2/(m+4))\widetilde O(n^{-2/(m+4)}) under dissipativity, where mm is the state dimension. A Girsanov-based policy-gradient identity and two synthetic experiments illustrate the generic and dissipative regimes.

Critical strengths

The paper gives matching localization bounds for a precisely specified SDE policy class. Its Girsanov-based experiments use a consistent policy parameterization, providing a coherent study of this restricted setting.

Critical weaknesses
  1. The statistical result supports a narrower setting than the claimed RL learning trade-off. Theorem 4.1 provides an oracle guarantee for policy search with a replayable transition simulator, shared random scenarios, and global empirical return maximization. This is a meaningful planning result. Its applicability to the experimental policy-gradient updates, total interaction budget, and cited diffusion RL methods remains unestablished. Moreover, the estimation upper bound alone does not establish an intrinsic statistical cost of greater expressivity. The closely related PEGASUS framework (Ng and Jordan, 2000) is also not discussed or cited.

  2. The expressivity analysis covers a restricted SDE policy class. The localization results address this class, and directly learning its drift is consistent with the Girsanov-based experiments. The class imposes uniform finite-KK drift regularity and persistent nondegenerate noise, without requiring correspondence to a forward noising process. Policies implementing reverse denoising dynamics can have extremely large or unbounded drift Lipschitz constants near the data endpoint. The paper does not distinguish these settings or establish how its drift budget characterizes terminal policy-distribution complexity. This limits the support for its broader diffusion-policy interpretation.

Questions for authors
  1. Would the authors explicitly restrict the statistical claims to simulator-based policy search, or establish their applicability to the cited RL learning methods?

  2. Which cited methods fit the analyzed SDE class, and would the authors restrict the diffusion-policy interpretation accordingly? If broader claims are retained, how does KK characterize terminal distribution complexity for policies tied to a forward noising process?

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer yTWV

We thank the reviewer for recognizing the matching localization bounds and the coherence of the SDE policy parameterization. We believe two distinctions clarify the scope and significance of our results: the distinction between a policy class and its training algorithm, and between a finite drift budget and an exact singular-endpoint limit.

1. Statistical guarantee, simulator access, and PEGASUS.

We thank the reviewer for highlighting the common-randomness construction and the connection to PEGASUS.. We emphasize that shared randomness is a coupling tool used in our proof/analysis. The coupling itself does not restrict the policy class or change the trajectory distribution induced by any individual policy.

Specifically, let Gf(ξ)G_f(\xi) denote the return obtained by applying policy ff to exogenous randomness ξ\xi. For every fixed ff, the resulting trajectory has exactly the distribution prescribed by the original MDP and policy, so E[Gf(ξ)]=V(f).\mathbb E[G_f(\xi)]=V(f). Using independent scenarios ξ1,…,ξn\xi_1,\ldots,\xi_n, we define V^n(f)=1n∑i=1nGf(ξi).\widehat V_n(f)=\frac1n\sum_{i=1}^nG_f(\xi_i).

Sharing these scenarios across policies specifies their joint empirical process while preserving each policy’s marginal trajectory law. This plays the same role as evaluating different candidate models on a common dataset in standard statistical ERM.

Our goal is to characterize the regret behavior of the optimal finite-sample policy. The nuance that makes the setup nontrivial is precisely that different policies will generate different distributions of actions and trajectories under the shared scenarios. Therefore, to compare between policies in a canonical way, we have to couple using the environment noise mechanism, which does not change with the policies.

We will clarify this role of shared randomness and discuss PEGASUS as a relevant antecedent. Our contribution is the quantitative approximation–estimation analysis of diffusion-policy ERM under this explicit, common-data formulation.

  1. The finite-Lipschitz SDE policy class suffices for near-optimal RL performance

The reviewer raises a question about the scope of our SDE policy class. Our main result is that for any prescribed positive performance tolerance, this class contains a policy that attains that tolerance with a finite drift Lipschitz budget, under the stated assumptions. The distinction lies in the objective. In diffusion-based distribution recovery, the goal is to reproduce a prescribed target law, and the exact reverse drift can have a diverging Lipschitz constant near the data endpoint when that law is singular. In RL, a diffusion policy is evaluated by its expected return. Lemma 3.1 constructs a policy with drift Lipschitz constant at most KK throughout the sampling interval and mean-squared localization error O(1/K)O(1/K) around an optimal action. Our value bounds show why this concentration suffices for near-optimal performance.

Regarding the reviewer’s concern about “how its drift budget characterizes terminal policy-distribution complexity,” our results characterize how tightly the terminal action law can concentrate under persistent noise. Proposition 3.3 gives a lower bound of order 1/K1/K for localization near a fixed finite set of actions, matching the order achieved by our construction. This lower bound holds for every admissible drift, including any reverse-denoising drift satisfying the hypotheses, so it requires no correspondence to a prescribed forward noising process. We will clarify that our sharp characterization concerns localization near optimal actions, the property that enters our RL value bounds.

AuthorsOpenReview ↗

Rebuttal for Reviewer yTWV (continued)

3. Connection to diffusion policies and reverse-denoising dynamics.

Our framework includes reverse-denoising SDEs whenever their drift, noise, and initialization satisfy the stated assumptions. This connection is structural: in the standard reverse-SDE formulation, the learned score enters the drift, while the noise schedule determines the diffusion coefficient. Most importantly, our localization lower bound applies generally to admissible reverse-denoising dynamics regardless of whether the drift was learned through score matching, policy gradients, or another procedure. Song et al., Section 3.2.

Requiring correspondence to a particular forward process is unnecessary for this lower bound. The theorem holds over the larger class of admissible KK-Lipschitz drifts, so it also holds over any reverse-score subclass contained within that class. Thus, the general SDE formulation gives a limitation that applies across different constructions of the drift.

Exact reverse-denoising dynamics can have an unbounded Lipschitz constant near the final sampling time. Our analysis asks how accurately diffusion policies can approximate optimal actions when the drift has a finite Lipschitz budget. For the RL objective, reproducing an ideal terminal distribution exactly is unnecessary: our value bounds show that sufficiently accurate concentration near optimal actions already yields nearly optimal performance.

Randomized policies play an established role in RL: they support exploration and allow the learner to retain multiple promising actions rather than commit prematurely to a single choice. Entropy-regularized methods such as Soft Actor-Critic Haarnoja et al., 2018 explicitly encourage policy stochasticity alongside reward maximization. This motivation is general to RL and is not specific to diffusion policies. Our persistent-noise setting is motivated by this broader role of randomization. The theoretical question is how accurately a diffusion policy can concentrate near optimal actions while its generating dynamics remain subject to nondegenerate noise. Our results quantify this relationship through the drift Lipschitz budget KK. As a concrete example of maintaining sampling noise in diffusion-based RL, DPPO imposes a positive minimum noise level during training to encourage exploration. DPPO, Section 4.3.

Finally, KK directly measures the regularity of the generating dynamics and controls a concrete aspect of the terminal distribution: its attainable concentration near optimal actions under nondegenerate noise. The admissible class permits nonlinear drifts and multimodal terminal laws. Our finite-sample analysis then connects this concentration benefit to statistical estimation, making KK a key parameter for studying approximation and learning jointly. We will clarify this interpretation without presenting KK as a complete measure of terminal-distribution complexity.

We hope that our responses and revisions have addressed your concerns. Please let us know if you have any further questions or comments before the discussion period ends; we would be happy to address them!

We believe the paper offers a new quantitative perspective on diffusion-policy learning by identifying the drift Lipschitz budget KK as a key parameter connecting approximation power and finite-sample learning guarantees. We are excited about the possibilities this perspective opens up, and hope the revised manuscript conveys its significance more clearly.

Reviewer_bqP3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper gives results on the universal approximation capabilities for learning diffusion policies in RL. The authors characterize a particular construction of the diffusion process which can approximate the optimal policy's value for any MDP, under reasonable regularity assumptions. They also complement these results with a matching lower bound. The authors further provide sample complexity results to learn the optimal policy within their constructed class, to optimally trade-off estimation and approximation errors, and validate their theoretical findings in simple experiments.

Critical strengths

The paper is well articulated, lays out an interesting problem. The questions are well contextualized relative to the literature. The assumptions are clearly stated, and the upper bound constructions and results seem novel within the stated framework. The presence of matching lower bounds gives further strength to the paper's conclusions. Finally, the full sample complexity bound trading off the estimation and approximation errors is also well constructed and completes the story, giving a full picture of learning with diffusion policies, at least in terms of the family constructed in this work.

Critical weaknesses

I found myself worrying about a few questions as I read this paper:

  1. The construction used seems quite different from the typical diffusion policies used in the literature. In particular, the mean drift is unimodal and any non-Gaussianity is obtained through the variance term, which I do not think is a valid caricature of practice. Consequently, the paper leaves the questions if these conclusions are relevant to practical use of these policies or not? Furthermore, if the classes used here are completely different from practice, do the findings from this work have any practical bearing on the design and use of diffusion policies in practice to improve their properties?

  2. While the results are interesting and important for diffusion policy learning, these are qualitatively not much different from standard non-parametric rates that one can obtain for learning the optimal policy in an MDP through other standard non-parametric classes, such as smoothness-based classes, wavelets etc. Can the authors contextualize their results wrt the broader literature on approximation and estimation in MDPs, and whether their constructions hint at any unique strengths of diffusion policy approaches wrt other universal approximators in theory?

Questions for authors

Please see the weaknesses above.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer bqP3

We thank the reviewer for recognizing the novelty of the upper-bound construction, the matching lower bound, and the finite-sample analysis. The two questions identify distinctions that we should make more explicit.

1. Relationship between the construction and practical diffusion policies.

The main goal of our paper is to characterize the fundamental approximation limits of Lipschitz-drift action diffusions in RL. Our general policy class allows nonlinear state-conditioned drifts. The (unimodal) linear mean-reverting drift is merely one construction for the approximation upper bound, asymptotic in the Lipschitz budget KK. We emphasize that the matching localization lower bound applies to all admissible KK-Lipschitz drifts under the current assumptions. It therefore also covers nonlinear drifts and multimodal terminal distributions. This broader scope is central to the result: more complicated drift geometry cannot improve the K−1K^{-1} order of squared localization within the specified class.

In other words, our fundamental limit theoretical results apply to stable practical diffusion policies, with respect to value function approximation in RL.

2. Relationship to classical nonparametric approximation and estimation.

Our contribution is to provide a statistical foundation connecting the approximation power of diffusion policies to their finite-sample learning guarantees. Approximation power must be assessed together with estimation error, and the drift Lipschitz budget KK is a key parameter linking these two aspects. Increasing KK allows more accurate approximation of optimal actions, while increasing the estimation term in our bound for generic drift classes. Our analysis quantifies this relationship and explains how the drift budget and representation capacity can be chosen according to the available sample size.

Establishing this connection requires understanding the diffusion sampler itself: how accurately can its terminal law concentrate near optimal actions when the drift has Lipschitz budget KK and the noise remains nondegenerate? We establish a sharp K−1K^{-1} squared-localization limit across the admissible drift class and give a construction attaining that order. Under the respective Bellman-gap assumptions, these localization results translate into value bounds. Classical approximation results for the optimal selector alone do not provide this characterization of the diffusion sampler.

Our finite-sample analysis then combines diffusion approximation, center realization, and estimation errors within a single learning guarantee. This makes explicit both the approximation benefit of a larger drift budget and its effect on the statistical bound, providing a quantitative basis for sample-aware diffusion-policy design.

We agree that the center-approximation and statistical-complexity terms have familiar nonparametric forms, and we do not claim a superior statistical rate over other universal approximators. These classical terms are ingredients of our analysis; the contribution is to connect them with a sharp characterization of diffusion concentration and quantify their joint implications for policy learning. Other center representations could also be considered through their corresponding approximation, complexity, and regularity properties.

We will strengthen the discussion of these classical connections and emphasize this joint approximation–estimation perspective more clearly.

We hope that our responses and revisions have addressed your concerns. Please let us know if you have any further questions or comments before the discussion period ends; we would be happy to address them!

We believe the paper offers a new quantitative perspective on diffusion-policy learning by identifying the drift Lipschitz budget KK as a key parameter connecting approximation power and finite-sample learning guarantees. We are excited about the possibilities this perspective opens up, and hope the revised manuscript conveys its significance more clearly.

Reviewer_bzJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the drift Lipschitz budget governs approximation and statistical learning in diffusion policies. Its contributions include matching a localization bounds under stated assumptions, finite-sample guarantees for neural drifts with improved rates under dissipativity, and a Girsanov-based policy-gradient formula.

My concern is that the finite-sample proof uses a maximal inequality, (C.34), that is invalid under population-L2L_2 covering assumption. As for experimental validation: the neural parameterization does not itself enforce the stated Lipschitz budget KK, and the theoretical guarantee for an exact empirical optimizer does not directly cover the implemented policy-gradient procedure in my understanding.

Critical strengths

The concentration bounds quantify the drift Lipschitz budget limits approximation of deterministic actions with noise. Their applicability across admissible drift architectures and sampling horizons gives the result relevance beyond a particular parameterization.

Critical weaknesses

Major issues

  1. The population-L2(P)L^2(P) covering assumption is insufficient for (C.34), leaving a gap in the proof of Theorem 4.1.
  2. Lemma D.6 invokes condition (D.5), whose uniform parameter regularity does not follow from Assumption D.1.
  3. The improved statistical rate under dissipativity lacks the required stability and uniform-convergence argument.
  4. The parameterization fθ,K=Ktanh⁡(MLPθ)f_{\theta,K}=K\tanh(\mathrm{MLP}_\theta) does not ensure Lip⁡a(fθ,K)≤K\operatorname{Lip}_a(f_{\theta,K})\le K

Minor issues

  1. The introduction should distinguish approximation of optimal value from representation of multimodal action distributions when discussing “expressivity.”
Questions for authors

My concern is that, the finite-sample guarantee relies on substantial regularity assumptions (and exact empirical optimization), while their connection to the learned policies is unclear. Could the authors clarify whether the experimental problems satisfy the assumed Lipschitz regularity of the optimal selector and quadratic Bellman-gap condition, whether training enforces the prescribed drift budget KK, and how optimization error affects guarantee’s applicability?

These points matter for assessing whether the observed dependence on KK supports the theoretical approximation.

7 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer bzJG

Weaknesses

Thank you for raising valid points regarding our theoretical proofs. We have submitted an revised manuscript that address these issues.

  1. Finite-sample maximal inequality. We agree that population L2L^2 covering alone does not justify the original inequality (C.34). We have replaced this step with a new argument in Appendix C.3 that controls how simulated returns change across policies and uses these stability estimates to establish a valid uniform-convergence bound. The finite-sample rate in Theorem 4.1 is preserved. This repair strengthens Assumption 4.2: when the environment noise is held fixed, the transition simulator must be Lipschitz in the state and action. This ensures that nearby inputs produce nearby next states under the shared simulation noise, providing the stability needed to compare policies. The condition holds, for example, for Lipschitz dynamics with additive noise.

  2. Parameter regularity in the policy-gradient proof. Appendix D has been re-written with a local Girsanov argument comparing policies with parameters θ\theta and θ+h\theta+h. The revised proof no longer invokes the previous Condition (D.5).

  3. Dissipative statistical rate. The revised proof now provides the missing stability and uniform-convergence argument. Let Zt=At−A~tZ_{t}=A_t-\tilde A_t be the difference of two action processes with the same Brownian motion coupling. The dissipativity condition allows us to control LqL_{q}-norms of ZtZ_{t} for every qq which gives uniform convergence. This in turn improves the statistical term from O~(sK/n)\tilde{O}(sK/n) to O~(s/n)\tilde{O}(s/n).

  4. Precise Lipschitz constant for experimental results. We have rescaled the Lipschitz constant to exactly KK in the experiment section of the updated manuscript, and the conclusions do not change.

AuthorsOpenReview ↗

Rebuttal for Reviewer bzJG (continued)

Questions for Authors

Thank you for raising the connection between the regularity assumptions, statistical estimation, and practical optimization. The assumptions on the optimal selector and Bellman gap capture a natural smooth-control regime with stable optimal decisions. They concern the underlying control problem and are not specific to diffusion policies.

Our analysis takes a statistical ERM perspective, separating the approximation and estimation properties of the diffusion-policy class from the convergence properties of a particular optimizer. Below, we give concrete sufficient conditions for these regularity assumptions and explain how optimization error enters the guarantee.

Regularity of the optimal selector.

Lipschitz regularity describes a regime in which small changes in state produce small changes in the optimal action. A concrete sufficient condition is that Q⋆Q^\star is jointly twice continuously differentiable near its optimal actions, with a unique maximizer g⋆(x)g^\star(x) lying in a common compact set, and

\qquad \|\nabla^2_{ax}Q^\star(x,g^\star(x))\|_{\mathrm{op}}\le B$$ uniformly in $x$, for some $\alpha>0$. Thus, our selector assumption is satisfied when optimal actions are uniformly stable under state perturbations. This is a standard sensitivity argument for solution mappings. [Dontchev and Rockafellar, *Implicit Functions and Solution Mappings*, Theorem 1B.1](https://sites.math.washington.edu/~rtr/papers/rtr234-Implicit2.pdf) **Quadratic Bellman gap.** The condition used in the finite-sample theorem is a quadratic *upper* bound. It follows directly from smoothness at an interior optimum: if $Q^\star(x,\cdot)$ has an $L_a$-Lipschitz gradient, then $\nabla_a Q^\star(x,g^\star(x))=0$, and Taylor’s inequality yields $$0\leq V^{\star}(x)-Q^\star(x,a) \le \frac{L_a}{2}\|a-g^\star(x)\|^2.$$ This implication does not require strong concavity. Uniform smoothness in a fixed neighborhood of the optimal actions also suffices when rewards are bounded: outside that neighborhood, boundedness of the value gap supplies the global quadratic bound with a larger constant. **The drift budget.** The prescribed $K$ is a constraint on the admissible drift class. In practice, it can be specified/enforced through a parameterization or norm constraints that guarantee the required action-Lipschitz bound throughout training. **Optimization error and ERM.** Exact empirical optimization is not essential to our statistical argument. For any admissible output drift $f_{\theta_t}$, define its empirical optimization error by $$\varepsilon_{\mathrm{opt}}(t)=\sup_{f\in\mathcal F_{K,s}}\widehat V_n(f) -\widehat V_n(f_{\theta_t}).$$ The same proof gives $$V^\star(\rho)-V(f_{\theta_t}) \le \widetilde O\!\left( K^{-1}+s^{-2/m}+\sqrt{sK/n}+sK/n \right) +\varepsilon_{\mathrm{opt}}(t).$$ Consequently, optimization to the statistical accuracy suffices to retain the stated learning rate. We emphasize that separating approximation, estimation, and optimization errors is an established approach in statistical learning. [Bottou and Bousquet, 2007](https://proceedings.neurips.cc/paper_files/paper/2007/file/0d3180d672e08b4c5312dcdafdf6ef36-Paper.pdf). As for optimization errors, the following condition is often adopted in literature to guarantee convergence to the ERM. For a concrete sufficient optimization condition, let $\widehat L_n(\theta)=-\widehat V_n(f_\theta)$. Suppose it is $L$-smooth and satisfies the Polyak–Łojasiewicz inequality $$\frac12\|\nabla\widehat L_n(\theta)\|^2 \ge \mu\bigl(\widehat L_n(\theta)-\widehat L_n^\star\bigr)$$ on a region preserved by gradient descent, with all iterates remaining admissible and $\widehat L_n^\star$ the global empirical optimum. Gradient descent with step size $1/L$ then satisfies $$\varepsilon_{\mathrm{opt}}(t) \le (1-\mu/L)^t\varepsilon_{\mathrm{opt}}(0).$$ Thus, the required empirical accuracy is reached at a geometric rate under these additional conditions. [Karimi, Nutini and Schmidt, 2016, Theorem 1](https://arxiv.org/pdf/1608.04636). Such conditions have substantive precedents in control: Fazel et al. establish gradient domination and global policy-gradient convergence for linear-quadratic regulation. Their result illustrates the role of optimization geometry, although it does not establish PL for general neural diffusion policies. [Fazel et al., 2018](https://proceedings.mlr.press/v80/fazel18a.html). Our paper focuses on the statistical trade-off without imposing this additional optimization geometry. Optimization remains important in practice, and the additive error term makes its effect on the guarantee explicit.

Thanks for the response. The clarification is useful, but I think my main concerns are still not fully addressed.

First, my concern with (C.34) remains. The revised proof establishes covering bounds under population L2(P)L^2(P), while the Dudley argument after symmetrization involves the empirical metric. I do not see how the added stability estimates justify this step and close the gap in Theorem 4.1.

The response also says that condition (D.5) has been removed. However, Lemma D.6 still uses Lemma D.4, which assumes (D.5). It is therefore still unclear how the required uniform parameter regularity follows from Assumption D.1.

For the dissipative rate, I could not find the additional proof described in the response. The experimental Lipschitz constraint is also unclear, since Ktanh⁡(MLPθ)K\tanh(\mathrm{MLP}_\theta) bounds the drift amplitude but does not itself ensure that the action-Lipschitz constant is at most KK.

Finally, I agree with Reviewer yTWV about the connection between the theorem and the experiments. The theorem uses empirical optimization over shared simulator scenarios, while the experiments use repeated policy-gradient updates with new samples. Having the correct marginal trajectory law for each policy does not address this difference. Also, an upper bound that increases with KK does not by itself show that the statistical cost necessarily increases with KK.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.