- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose a novel deterministic policy gradient (DPG) algorithm called AdvG using multi-step temporal difference error for value estimation while learning the advantage rate’s action derivative from individual exploratory perturbations. By analyzing critic stability, the authors establish the first finite-sample joint actor and critic convergence guarantee in model-free DPG learning for nonlinear controlled diffusions.
Critical strengths
-
The authors propose a novel algorithm and establish a finite-sample joint actor–critic convergence guarantee for model-free deterministic policy gradient learning in nonlinear controlled diffusion systems.
-
The authors compare the proposed algorithm against established reinforcement learning methods, including SAC, TD3, and DDPG. This comparision helps in understanding the relative performance of the algorithm.
Critical weaknesses
- The experiments are performed on LQ control and noisy pendulum tasks. These are relatively simple environments and hence the experimental results may not provide any information on the effectiveness in complex, high-dimensional real-world applications.
Questions for authors
Please refer to the weakness section.
3 comments
the authors have addressed my concerns. I maintain my score
I thank the authors for addressing my concerns. I have decided to retain my score.
Official Response to Reviewer MAnv
We thank you for the positive assessment of the algorithm and convergence analysis.
W1: The experiments on LQ control and noisy Pendulum provide limited evidence for complex, high-dimensional real-world applications.
A1: To address this concern, we added experiments on higher-dimensional tasks including HalfCheetah and Hopper from Gymnasium, comparing AdvG with CT-DDPG at different observation intervals. The revised manuscript presents the learning curves in Appendix E.1.3, Figures 10-11, and the numerical results in Table 3.
The reward curves show that AdvG learns faster than CT-DDPG, and AdvG achieves higher final mean returns at both observation intervals on each task. The final returns are:
These results extend the evidence for AdvG beyond LQ and noisy Pendulum to more complex, higher-dimensional simulated control.