- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the theoretical mechanics of continuous-time diffusion policy learning as the physical time step size approaches zero (). The authors identify a fundamental action-gradient attenuation phenomenon, proving that under stationary action sampling, the Q-function gradient signal degrades at , stalling policy updates. They analyze the standard practice of naive scalar calibration (e.g., scaling by ) and demonstrate both theoretically and analytically (via a Linear-Quadratic model) that scalar scaling fails to restore the true policy improvement direction due to multi-mode action relaxation, while amplifying critic estimation noise () and causing iterative parameter divergence (). To characterize the correct update, the paper introduces the Poisson Improvement Field (), derived from the Poisson equation, as the target for true policy improvement.
Critical strengths
1.Rigor and Depth of Theoretical Analysis: The theoretical formulation of action-gradient attenuation ( decay, Theorem 3.2) and the mathematical derivation of the Poisson Improvement Field (Theorem 4.1) offer a deep, foundational understanding of continuous-time diffusion RL dynamics.
2.Insightful Diagnostic on Scalar Calibration: The LQ analysis in Section 5 elegantly exposes the "scalar calibration trap" (Proposition 5.1), proving that even when a scalar fix improves performance in a single step (), maintaining a constant scalar across policy iterations alters relaxation modes and leads to parameter explosion.
3.Identify Crucial Pitfalls in High-Frequency Control: The identification of critic noise amplification (, Corollary 4.3) under small step sizes provides important theoretical clarity for practitioners attempting to scale diffusion policies to high-frequency or continuous-time control.
Critical weaknesses
1.Disconnect Between Theory and Main-Text Empirical Presentation: While Appendix E contains an extensive collection of experimental results, the empirical evaluation in the main text (Section 6 and Figure 2) lacks clarity and narrative alignment with the theory. Figure 2(a) shows a severe performance collapse for naive scalar calibration () at . While this plot functions as a negative diagnostic illustrating noise amplification (), the main text fails to cleanly contextualize it, leaving the reader confused as to whether the experiment represents an algorithmic failure or a diagnostic proof.
2.Lack of a Practical, Deployable Algorithm: The paper successfully diagnoses the failure of existing methods and introduces the Poisson Improvement Field () as the theoretical gold standard. However, it stops short of providing an efficient, scalable algorithm to compute or approximate in high-dimensional continuous control tasks without incurring prohibitive computational costs.
3.Presentation and Narrative Flow: A significant portion of the empirical evidence that could clarify the paper's claims is deferred to the Appendix, leaving Section 6 feeling incomplete. The paper reads primarily as a diagnostic/theoretical paper, but markets itself partially around empirical normalization strategies without fully bridging the gap.
Questions for authors
-
Practical Realization of : Given that computing the true Poisson Improvement Field requires solving an Poisson equation over relaxation dynamics, what do the authors view as the primary bottlenecks for developing a computationally scalable online approximation in high-dimensional tasks?
-
Reorganization of Empirical Evidence: Would the authors consider moving some key explanatory experiments from Appendix E into Section 6 to better bridge the gap between the theoretical proofs and empirical observations in the main body?
-
High-Frequency Control vs. Macro-Action Paradigm: From a broader control and decision-making perspective, pushing forces the agent to perform ultra-high-frequency closed-loop action updates, leading to the severe signal attenuation and noise amplification identified in your work. Diffusion policies in robotics often excel precisely because they generate extended temporal trajectories (action chunking / macro-actions) rather than instantaneous step-by-step updates. Do the authors view ultra-high-frequency Q-gradient updates as a long-term viable path for continuous-time diffusion RL, or does your diagnostic work suggest that the community should shift toward hierarchical / event-driven architectures that operate over coarser time scales?
4 comments
(3) Insufficient empirical validation: The main text does not sufficiently connect the theoretical findings with the experiments, while important empirical evidence is deferred to the appendix.
We thank the reviewer for this helpful comment. We have revised Section 6 to strengthen the connection between the theoretical results and empirical observations. Specifically, Figure 2 (Page 9) verifies the predicted attenuation using an exact population critic, while the learned-critic results in Appendix E.1 (Table 3, Page 28) examine whether the same behavior can be observed with an estimated critic. We also include a scalar-calibration sensitivity study (Table 1, Page 9) to demonstrate that restoring the gradient magnitude does not guarantee improved performance for every calibration coefficient. Figure 3 (Page 10) further illustrates the effects of actor normalization, and Figure 4 (Page 10) evaluates adaptive magnitude calibration on MuJoCo tasks. We have also clarified that the primary contribution is the theoretical analysis of time-scale mismatch and Poisson-based policy improvement, while the normalization experiments illustrate its practical implications rather than establish a general implementation of the Poisson improvement field.
(4) Scalability of the Poisson improvement field: What are the main computational challenges in developing an efficient online approximation of for high-dimensional control tasks?
We identify three main challenges. First, the Poisson corrector is defined over the joint state–action space, making its representation and evaluation challenging in high dimensions. Second, the approximation must capture different action-relaxation rates. As shown in Proposition 4.2 (Page 6), a single scalar normalization cannot generally reproduce the contributions of multiple active relaxation modes. Third, online approximation must remain stable as the policy evolves, since changes in the policy alter the action dynamics and the corresponding Poisson corrector. Moreover, critic-gradient estimation errors can be amplified by small-step normalization, as quantified in Corollary B.1 (Page 21). A possible direction is to approximate using a neural network trained with sampled trajectories and a Poisson-equation residual. However, controlling approximation errors and maintaining stable online updates remain open challenges. We view this as future work.
(5) High-frequency control versus macro-actions: Does the analysis imply that ultra-high-frequency critic-guided updates should be avoided in favor of coarser, hierarchical, or event-driven policy updates?
Thank you for raising this important point. We do not view ultra-high-frequency Q-gradient updates as a universally desirable long-term solution. Our analysis is diagnostic rather than prescriptive: under stationary action resampling, the per-step critic signal decreases as , while dividing it by can amplify critic estimation errors.
Importantly, physical control frequency and policy-learning frequency need not be identical. An agent can maintain frequent environment feedback while updating its policy at a coarser timescale, or employ action chunks and event-driven updates. Our results therefore do not rule out high-frequency control or mandate hierarchical architectures. Instead, they highlight the importance of choosing policy-update frequencies according to the strength and reliability of critic guidance. We have discussed multi-timescale and event-driven policy updates as promising directions for future work in the conclusion (Page 10).
Thank you for your sincere answers to my comments and questions. My understanding and appreciation of this research have grown. While the authors' responses and revisions were satisfactory, I will keep the original score (Weakly Accept : 3) for the following reasons: (1) a scalable, practical algorithm to approximate the full Poisson improvement field remains open for future work, and (2) as the authors candidly acknowledge in their response, pushing decision-making frequency toward the limit is inherently fraught with structural bottlenecks and may not be the ultimate path to continuous-time control compared to coarser, multi-timescale, or event-driven policy architectures.
Thank you very much for your thoughtful feedback. We sincerely appreciate your recognition of our revisions and the time you have devoted to reviewing our paper. Your constructive feedback has been very valuable in helping us improve the manuscript and identify directions for future research.
We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate your positive comments, such as: "Rigor and Depth of Theoretical Analysis". In the following, we provide detailed responses to the comments point by point.
(1) Disconnect between theory and experiments: The experimental results, particularly the performance degradation under λA=h at small h, are not sufficiently connected to the theoretical analysis. It is unclear whether this reflects an algorithmic failure or a diagnostic result illustrating noise amplification.
We thank the reviewer for identifying this ambiguity. Figure 3 illustrates the practical limitations of naive unit-coefficient normalization (λA=h) in joint actor–critic training, rather than demonstrating a universally effective algorithm. The theoretical distinction is that dividing the critic gradient by h compensates for its deterministic O(h) attenuation but also amplifies critic-gradient estimation errors by 1/h, as quantified in Corollary B.1 (Appendix B, Page 21). The experiment illustrates the resulting practical trade-off. At h=0.01, normalization substantially increases the actor-target magnitude (Figure 3(b), Page 10). Although the raw discounted reward improves, the control cost increases substantially, resulting in a lower full discounted value (Appendix E.4, Table 5, Page 31). Thus, this experiment does not indicate a failure of the underlying control problem or the exact Poisson improvement field. Rather, it shows that restoring the gradient magnitude alone does not guarantee stable repeated actor–critic updates. We have clarified this interpretation in Section 6.1 (Pages 8–9).
(2) Lack of a practical algorithm: The Poisson improvement field is theoretically justified, but the paper does not provide an efficient and scalable way to approximate it in high-dimensional control tasks.
We thank the reviewer for raising this important concern. We agree that directly computing the Poisson improvement field BΨ=λ0−1∇aUΨ can be computationally demanding in high-dimensional continuous control, and that an efficient, scalable approximation remains an open problem. As a practical step, we have added an adaptive magnitude calibration method and evaluated it on MuJoCo locomotion tasks (Section 6.2, Pages 9–10). Specifically, it tracks the running magnitude of the time-normalized learned critic gradient:
g_n=\alpha_1g_{n-1}+(1-\alpha_1)\mathbb E[\|\widetilde g_i\|],\quad \Psi_{\mathrm{tar},i}=\frac{\widetilde g_i}{\lambda_0g_n},$$ where $\alpha_1=0.99$. This requires only a lightweight online statistic and does not involve explicitly solving a Poisson equation. Importantly, it calibrates the guidance magnitude rather than approximating the full Poisson improvement field. The MuJoCo results (Figure 4, Page 10) show improved training stability and returns over the fixed-coefficient baseline under the tested irregular decision schedules. This is a practical magnitude-calibration strategy, not a scalable implementation of $B^\Psi$. Developing such an approximation remains an important direction for future work.