acceptodds
Under review as a conference paper at ICLR 2027

Time Scales in Diffusion Policy Learning: Action-Gradient Attenuation and Poisson-Based Policy Improvement

Abstract

Continuous-time diffusion reinforcement learning involves two distinct time scales: the evolution of the environment and the dynamics used to sample actions. We show that a mismatch between these time scales changes the critic signal used to update the actor. Under stationary action resampling, the critic action-gradient signal vanishes linearly with the physical time step \(h\), leading to increasingly weak actor updates as \(h\) tends to 0. We derive a finite-horizon corrector that connects this finite-step effect to the stationary-resampling and fast-action limits, and characterize the continuous-time policy-improvement direction through a performance-difference identity based on the Poisson equation. Normalizing the finite-step actor signal by \(h\) restores its magnitude, but recovers the limiting policy-improvement direction only when the relevant action modes share a common relaxation rate. An exactly solvable linear–quadratic (LQ) model further shows that a fixed scalar normalization can improve the first policy update yet cause the policy updates to diverge when applied repeatedly. Experiments with both exact and learned critics validate the predicted linear attenuation and show that normalization by \(h\) improves learning at small physical time steps, while performance remains sensitive to the choice of scalar coefficient.

Primary area:
reinforcement learning
Keywords:
Continuous-Time Reinforcement Learning, Diffusion Policies, Policy Improvement, Stochastic Differential Equations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pnTuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the theoretical mechanics of continuous-time diffusion policy learning as the physical time step size approaches zero (h→0h \to 0). The authors identify a fundamental action-gradient attenuation phenomenon, proving that under stationary action sampling, the Q-function gradient signal degrades at O(h)O(h), stalling policy updates. They analyze the standard practice of naive scalar calibration (e.g., scaling by 1/h1/h) and demonstrate both theoretically and analytically (via a Linear-Quadratic model) that scalar scaling fails to restore the true policy improvement direction due to multi-mode action relaxation, while amplifying critic estimation noise (ηh/h\eta_h / h) and causing iterative parameter divergence (κk→∞\kappa_k \to \infty). To characterize the correct update, the paper introduces the Poisson Improvement Field (BΨB^\Psi), derived from the Poisson equation, as the target for true policy improvement.

Critical strengths

1.Rigor and Depth of Theoretical Analysis: The theoretical formulation of action-gradient attenuation (O(h)O(h) decay, Theorem 3.2) and the mathematical derivation of the Poisson Improvement Field (Theorem 4.1) offer a deep, foundational understanding of continuous-time diffusion RL dynamics.

2.Insightful Diagnostic on Scalar Calibration: The LQ analysis in Section 5 elegantly exposes the "scalar calibration trap" (Proposition 5.1), proving that even when a scalar fix improves performance in a single step (k=1k=1), maintaining a constant scalar across policy iterations alters relaxation modes and leads to parameter explosion.

3.Identify Crucial Pitfalls in High-Frequency Control: The identification of critic noise amplification (ηh/h\eta_h / h, Corollary 4.3) under small step sizes hh provides important theoretical clarity for practitioners attempting to scale diffusion policies to high-frequency or continuous-time control.

Critical weaknesses

1.Disconnect Between Theory and Main-Text Empirical Presentation: While Appendix E contains an extensive collection of experimental results, the empirical evaluation in the main text (Section 6 and Figure 2) lacks clarity and narrative alignment with the theory. Figure 2(a) shows a severe performance collapse for naive scalar calibration (λA=h\lambda_A = h) at h=0.01h = 0.01. While this plot functions as a negative diagnostic illustrating noise amplification (ηh/h\eta_h / h), the main text fails to cleanly contextualize it, leaving the reader confused as to whether the experiment represents an algorithmic failure or a diagnostic proof.

2.Lack of a Practical, Deployable Algorithm: The paper successfully diagnoses the failure of existing methods and introduces the Poisson Improvement Field (BΨB^\Psi) as the theoretical gold standard. However, it stops short of providing an efficient, scalable algorithm to compute or approximate BΨB^\Psi in high-dimensional continuous control tasks without incurring prohibitive computational costs.

3.Presentation and Narrative Flow: A significant portion of the empirical evidence that could clarify the paper's claims is deferred to the Appendix, leaving Section 6 feeling incomplete. The paper reads primarily as a diagnostic/theoretical paper, but markets itself partially around empirical normalization strategies without fully bridging the gap.

Questions for authors
  1. Practical Realization of BΨB^\Psi: Given that computing the true Poisson Improvement Field BΨB^\Psi requires solving an Poisson equation over relaxation dynamics, what do the authors view as the primary bottlenecks for developing a computationally scalable online approximation in high-dimensional tasks?

  2. Reorganization of Empirical Evidence: Would the authors consider moving some key explanatory experiments from Appendix E into Section 6 to better bridge the gap between the theoretical proofs and empirical observations in the main body?

  3. High-Frequency Control vs. Macro-Action Paradigm: From a broader control and decision-making perspective, pushing h→0h \to 0 forces the agent to perform ultra-high-frequency closed-loop action updates, leading to the severe O(h)O(h) signal attenuation and O(1/h)O(1/h) noise amplification identified in your work. Diffusion policies in robotics often excel precisely because they generate extended temporal trajectories (action chunking / macro-actions) rather than instantaneous step-by-step updates. Do the authors view ultra-high-frequency Q-gradient updates as a long-term viable path for continuous-time diffusion RL, or does your diagnostic work suggest that the community should shift toward hierarchical / event-driven architectures that operate over coarser time scales?

4 comments

We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate your positive comments, such as: "Rigor and Depth of Theoretical Analysis". In the following, we provide detailed responses to the comments point by point.

(1) Disconnect between theory and experiments: The experimental results, particularly the performance degradation under λA=h\lambda_A=h at small hh, are not sufficiently connected to the theoretical analysis. It is unclear whether this reflects an algorithmic failure or a diagnostic result illustrating noise amplification.

We thank the reviewer for identifying this ambiguity. Figure 3 illustrates the practical limitations of naive unit-coefficient normalization (λA=h\lambda_A=h) in joint actor–critic training, rather than demonstrating a universally effective algorithm. The theoretical distinction is that dividing the critic gradient by hh compensates for its deterministic O(h)O(h) attenuation but also amplifies critic-gradient estimation errors by 1/h1/h, as quantified in Corollary B.1 (Appendix B, Page 21). The experiment illustrates the resulting practical trade-off. At h=0.01h=0.01, normalization substantially increases the actor-target magnitude (Figure 3(b), Page 10). Although the raw discounted reward improves, the control cost increases substantially, resulting in a lower full discounted value (Appendix E.4, Table 5, Page 31). Thus, this experiment does not indicate a failure of the underlying control problem or the exact Poisson improvement field. Rather, it shows that restoring the gradient magnitude alone does not guarantee stable repeated actor–critic updates. We have clarified this interpretation in Section 6.1 (Pages 8–9).

(2) Lack of a practical algorithm: The Poisson improvement field is theoretically justified, but the paper does not provide an efficient and scalable way to approximate it in high-dimensional control tasks.

We thank the reviewer for raising this important concern. We agree that directly computing the Poisson improvement field BΨ=λ0−1∇aUΨB^\Psi=\lambda_0^{-1}\nabla_a U^\Psi can be computationally demanding in high-dimensional continuous control, and that an efficient, scalable approximation remains an open problem. As a practical step, we have added an adaptive magnitude calibration method and evaluated it on MuJoCo locomotion tasks (Section 6.2, Pages 9–10). Specifically, it tracks the running magnitude of the time-normalized learned critic gradient:

g_n=\alpha_1g_{n-1}+(1-\alpha_1)\mathbb E[\|\widetilde g_i\|],\quad \Psi_{\mathrm{tar},i}=\frac{\widetilde g_i}{\lambda_0g_n},$$ where $\alpha_1=0.99$. This requires only a lightweight online statistic and does not involve explicitly solving a Poisson equation. Importantly, it calibrates the guidance magnitude rather than approximating the full Poisson improvement field. The MuJoCo results (Figure 4, Page 10) show improved training stability and returns over the fixed-coefficient baseline under the tested irregular decision schedules. This is a practical magnitude-calibration strategy, not a scalable implementation of $B^\Psi$. Developing such an approximation remains an important direction for future work.

(3) Insufficient empirical validation: The main text does not sufficiently connect the theoretical findings with the experiments, while important empirical evidence is deferred to the appendix.

We thank the reviewer for this helpful comment. We have revised Section 6 to strengthen the connection between the theoretical results and empirical observations. Specifically, Figure 2 (Page 9) verifies the predicted O(h)O(h) attenuation using an exact population critic, while the learned-critic results in Appendix E.1 (Table 3, Page 28) examine whether the same behavior can be observed with an estimated critic. We also include a scalar-calibration sensitivity study (Table 1, Page 9) to demonstrate that restoring the gradient magnitude does not guarantee improved performance for every calibration coefficient. Figure 3 (Page 10) further illustrates the effects of actor normalization, and Figure 4 (Page 10) evaluates adaptive magnitude calibration on MuJoCo tasks. We have also clarified that the primary contribution is the theoretical analysis of time-scale mismatch and Poisson-based policy improvement, while the normalization experiments illustrate its practical implications rather than establish a general implementation of the Poisson improvement field.

(4) Scalability of the Poisson improvement field: What are the main computational challenges in developing an efficient online approximation of BΨB^\Psi for high-dimensional control tasks?

We identify three main challenges. First, the Poisson corrector UΨ(x,a)U^\Psi(x,a) is defined over the joint state–action space, making its representation and evaluation challenging in high dimensions. Second, the approximation must capture different action-relaxation rates. As shown in Proposition 4.2 (Page 6), a single scalar normalization cannot generally reproduce the contributions of multiple active relaxation modes. Third, online approximation must remain stable as the policy evolves, since changes in the policy alter the action dynamics and the corresponding Poisson corrector. Moreover, critic-gradient estimation errors can be amplified by small-step normalization, as quantified in Corollary B.1 (Page 21). A possible direction is to approximate UΨU^\Psi using a neural network trained with sampled trajectories and a Poisson-equation residual. However, controlling approximation errors and maintaining stable online updates remain open challenges. We view this as future work.

(5) High-frequency control versus macro-actions: Does the analysis imply that ultra-high-frequency critic-guided updates should be avoided in favor of coarser, hierarchical, or event-driven policy updates?

Thank you for raising this important point. We do not view ultra-high-frequency Q-gradient updates as a universally desirable long-term solution. Our h→0h\to0 analysis is diagnostic rather than prescriptive: under stationary action resampling, the per-step critic signal decreases as O(h)O(h), while dividing it by hh can amplify critic estimation errors.

Importantly, physical control frequency and policy-learning frequency need not be identical. An agent can maintain frequent environment feedback while updating its policy at a coarser timescale, or employ action chunks and event-driven updates. Our results therefore do not rule out high-frequency control or mandate hierarchical architectures. Instead, they highlight the importance of choosing policy-update frequencies according to the strength and reliability of critic guidance. We have discussed multi-timescale and event-driven policy updates as promising directions for future work in the conclusion (Page 10).

Thank you for your sincere answers to my comments and questions. My understanding and appreciation of this research have grown. While the authors' responses and revisions were satisfactory, I will keep the original score (Weakly Accept : 3) for the following reasons: (1) a scalable, practical algorithm to approximate the full Poisson improvement field BΨB^\Psi remains open for future work, and (2) as the authors candidly acknowledge in their response, pushing decision-making frequency toward the h→0h \to 0 limit is inherently fraught with structural bottlenecks and may not be the ultimate path to continuous-time control compared to coarser, multi-timescale, or event-driven policy architectures.

Thank you very much for your thoughtful feedback. We sincerely appreciate your recognition of our revisions and the time you have devoted to reviewing our paper. Your constructive feedback has been very valuable in helping us improve the manuscript and identify directions for future research.

Reviewer_tpwSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the mismatch between the physical time scale hh and the action-sampling time scale τ\tau in continuous-time RL with diffusion policy. The theoretical results include:

  1. Under stationary action resampling, the action-dependent critic signal decreases linearly with hh.
  2. A Poisson improvement field that guarantees policy improvement.
  3. The effectiveness and limitations of scalar calibration. These results are verified in an solvable LQ model and experiments.
Critical strengths
  1. The paper studies an interesting and important setting.
  2. The theoretical analysis is valid and insightful.
Critical weaknesses

None.

Questions for authors

How is the actor implemented and updated in the experiments? If I do not misunderstand, the actor is updated for every physical time step, so the total number of updates increases as hh decreases. Does this compansate the linearly decreasing action-gradient signal? What is the effect of hh to the training dynamics of the actor?

1 comment

We thank the reviewer for raising this important question. In the LQ experiments, the actor is parameterized as an affine drift field, Ψθ(x,a)=−κ(a−px−d)\Psi^\theta(x,a)=-\kappa(a-px-d), and is updated once per physical decision step. Thus, over a fixed physical training horizon TT, the number of updates scales as Nh=T/hN_h=T/h.

We agree that the increased update frequency can partially compensate for the O(h)O(h) attenuation of the per-step critic action gradient. Under an idealized setting with aligned update directions, the cumulative update can remain O(1)O(1). However, this does not guarantee equivalent training dynamics in practice, since the actor and critic co-evolve, and stochastic gradient estimation and optimizer effects can alter successive updates.

Our experiments illustrate this distinction. Under fixed normalization, the actor-target magnitude decreases from 1.437 at h=0.1h=0.1 to 0.495 at h=0.01h=0.01, while the final environment reward drops from 0.227 to −4.167-4.167 (Appendix E.2, Table 4, Page 29). Step-dependent normalization improves the reward at smaller hh, but excessive guidance can increase the control cost and degrade the full discounted value. We have clarified the relationship between update frequency, per-step signal magnitude, and practical training dynamics in Section 6.1 (Page 9).

Reviewer_jZ4MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am putting my entire review here, as a sort of protest against the review form, which just makes it harder to be clear, in my honest opinion.

Paper 31306: Time scales in diffusion policy learning: Action-gradient attenuation and poisson-based policy improvement

This paper presents new theoretical and empirical results in diffusion policy learning. This is a setting involving continuous time, state, and action. The results concern the inter-relationships between the physical decision interval and the action-sampling horizon, and their impact on the effectiveness of the critic.

Unfortunately, I found this paper impossible to understand. I found no hint in this paper as to what "diffusion policy learning" is, or as to what either of the two central time quantities are. The latter are the "physical decision interval" h and the "action-sampling horizon" \tau. The names of these quantities are suggestive of what they might mean but, surprisingly, there is never any longer explanation or definition. The same is true for the "critic action-gradient signal". There is considerable advanced mathematical notation in the paper, so some explanation in ordinary language is needed, but not provided. I did not understand the setting, the claims, or the proofs of the claims. Undoubtedly, my lack of understanding is partially my fault; this is not an area in which I have a strong background. But I can see many ways in which the authors failed to communicate well, as I detail below.

Given all this, what am I to recommend about acceptance? The bottom line is that I can not recommend accepting a paper that can not even express its claims to contribution in a recognizable form. This may be good work, but it is not presented in a way that reveals it. With what is in the paper, I do not think it is possible to understand. In the following I identify areas where the the presentation is lacking such that it defies understanding.


The idea of "diffusion policy learning" is discussed but not explained in the ending parts of the first two introductory paragraphs. No reference is given for this idea. There are references given in the related work section on "Diffusion and score-based policy learning" but this section seems to be about how the setting in this paper is different. So then the setting should be explained in this paper. But the next section is not explanatory. It continues to presume that the ideas are already clear. For example, it says that this section "specifies the small-step scaling of the stationary-resampling critic action gradient". The use of the definite article "the" foreshadows that the meaning of these terms will never be explained; it is presumed. This section descends immediately into dense mathematical notation without explanation. Are the definitions here really self explanatory? I don't think so. Is it okay to not say what half of the symbols mean? Such as b_X, \sigma_X, \Psi, f, \cal A, ... Apparently, you are presuming that the reader can guess a lot from context, like which objects are scalars and which are functions, vectors, matrices, or operators, or which which object corresponds to reward. There is no distinction between mathematically defined things are their estimates. I am sorry, I don't think there is enough said in Section 3 for any of the mathematics to make sense, much less be unambiguous. And I don't see any attempt to make them make sense. The authors need to accept an obligation to go to considerable lengths to make their ideas accessible, to explain, but I see no such effort here.

In these sections, there is also no explanation of the other two key concepts, the "physical decision interval" h and the "action-sampling horizon" \tau. The idea of "re-sampling" is also not made clear. This can't be so hard to explain! What is it?! Readers should not have to guess!

Small point: the paper seems unaware of the prior work by Jae-young Lee at the University of Manchester, published in Automatica (2012, 2014, 2021) and other journals. It is a common name, so see https://scholar.google.ca/citations?user=Zh7PeGUAAAAJ&hl=en&oi=sra

Critical strengths

I don't find this part of the review form helpful.

Critical weaknesses

see review above.

Questions for authors

see review above.

2 comments

We sincerely thank your time and efforts in reviewing our paper. In the revised manuscript, we have substantially improved the presentation and organization. Specifically, we have made the following improvements:

(1) Lack of explanation and references for diffusion policy learning.

We thank the reviewer for pointing this out. We have revised the Introduction (Page 1) to briefly explain how diffusion policies generate actions through progressive denoising and how the critic's action gradient guides policy learning. We have also expanded the related-work discussion (Page 2) and added relevant references on diffusion models, diffusion policies, Q-Score Matching, and Continuous Q-Score Matching, including Ho et al. (2020), Chi et al. (2025), Psenka et al. (2024), and Hua et al. (2025).

(2) Unclear definitions of the two time scales and action resampling.

We agree that these quantities are central to our setting. We have added explicit definitions in the Introduction (Page 1): hh denotes how long an action is applied to the environment, while τ\tau denotes how long the action diffusion evolves at a fixed physical state to generate the next action. To make these two time scales and their interaction easier to follow, we have also added Figure 1 (Page 2), which provides an intuitive illustration of the physical evolution and action-sampling processes.

In Section 3.1 (Page 4), we distinguish finite-horizon sampling, where the next action retains dependence on the previous action, from stationary resampling, where it is drawn directly from the invariant conditional distribution. We also formalize these two cases using the corresponding transition operators.

(3) Unclear meaning of the critic action-gradient signal.

We have clarified this concept in the Introduction (Page 1), explaining that the critic action gradient measures how the predicted return changes with the action and provides guidance for updating the policy. In Section 3.2 (Page 5), we further define its magnitude and establish the O(h)O(h) attenuation under stationary action resampling. We also added an intuitive explanation following Theorem 3.2: the action-dependent component of the critic accumulates over a physical interval of length hh, resulting in a vanishing action gradient as hh decreases.

(4) Dense mathematical notation and insufficient explanations.

We acknowledge that the original presentation introduced mathematical notation too quickly. We have revised Section 3.1 (Pages 3–4) to clarify the state and action spaces, the roles of the state drift bXb_X, diffusion coefficient σX\sigma_X, and policy drift Ψ\Psi, as well as the corresponding state and action generators. We also explain the invariant action distribution, averaging operator, and transition operators before using them in the subsequent analysis.

Furthermore, we have improved the intuitive explanations in Sections 3–5 to clarify the roles of the advantage rate, Poisson corrector, and policy-improvement field, and to explain how these concepts are connected to the main theoretical results.

(5) Missing related work by Jae-young Lee and coauthors.

Thank you for pointing us to these important works. We have expanded the related-work discussion (Page 3) to include Lee et al. (2012, 2014, 2015) and Lee and Sutton (2021), covering their contributions to integral Q-learning and policy iteration for continuous-time systems. All corresponding references have been added. We also clarify that, while these works develop continuous-time reinforcement learning and policy iteration methods, our work focuses on the interaction between physical decision intervals and diffusion-based action sampling, and its effects on critic guidance and policy improvement.

References

Jae Young Lee, Jin Bae Park, and Yoon Ho Choi. Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems. Automatica, 48(11):2850–2859, 2012.

Jae Young Lee, Jin Bae Park, and Yoon Ho Choi. On integral generalized policy iteration for continuous-time linear quadratic regulations. Automatica, 50(2):475–489, 2014.

Jae Young Lee, Jin Bae Park, and Yoon Ho Choi. Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations. IEEE Transactions on Neural Networks and Learning Systems, 26(5):916–932, 2015.

Jae Young Lee and Richard S. Sutton. Policy iterations for reinforcement learning problems in continuous time and space: Fundamental theory and methods. Automatica, 126:109421, 2021.

Dear Reviewer jZ4M,

Thank you again for your review. As the rebuttal period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses to your comments. We hope that our clarifications have adequately addressed your concerns. If so, we would be grateful if you could consider updating your rating accordingly. If you have any remaining concerns or questions, please feel free to let us know, and we would be happy to address them.

Comments on the paper

2 comments

Dear Reviewers,

We kindly invite you to review our rebuttal and follow-up responses. We would greatly appreciate any final feedback and are happy to address any remaining concerns before the discussion closes.

Thank you again for your time!

We sincerely thank the reviewers and the Area Chair for their constructive feedback and the opportunity to improve our manuscript. We would like to briefly summarize both the main contributions of our work and the improvements made during the rebuttal.

Main contributions. This paper investigates the interaction between two time scales in continuous-time diffusion policy learning: the physical decision interval hh and the action-sampling horizon τ\tau. Our theoretical analysis establishes that, under stationary action resampling, the critic action gradient attenuates at an O(h)O(h) rate. We further derive a finite-horizon corrector, establish an exact performance-difference identity, and characterize a Poisson-based policy-improvement field that accounts for the relaxation dynamics of the action process. We show that scalar normalization can recover the correct improvement direction only when all active action modes share a common relaxation rate. Experiments with exact and learned critics, together with LQ and MuJoCo results, validate the predicted attenuation and illustrate both the benefits and limitations of step-dependent magnitude calibration.

Revisions and improvements. In response to the reviewers' concerns, we have improved the manuscript in three aspects. First, we clarified the problem setting, explicitly defined the two time scales, added an illustrative figure, improved the explanations of key mathematical concepts. Second, we strengthened the connection between theory and experiments by clarifying the O(h)O(h) signal attenuation, its interaction with actor update frequency, and the limitations of scalar normalization. Third, we added adaptive magnitude calibration (AMC) experiments on MuJoCo locomotion tasks with irregular decision intervals.

We appreciate that Reviewer pnTu provided a follow-up response, explicitly acknowledging that our answers and revisions improved their understanding and appreciation of the research, and describing our responses and revisions as satisfactory. We believe that the revised manuscript presents a clearer and more complete account of its theoretical contributions and empirical implications.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.