- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
On-policy algorithms update the policy using the value function of the previous policy, which may cause the value-lag issue. However, learning a separate new value function increases the computation burden. This paper proposes an approach that leverages the first-order information of the value network to correct value estimates without additional rollout or learnable parameters.
Critical strengths
This paper studies an interesting and important problem in on-policy RL. Its presentation is clear and easy-to-follow. I did not check every detail of its theory proofs, but their claims seem reasonable to me. Empirical results are comprehensive and look good.
Critical weaknesses
Figure 5 compares the performance of different methods on MuJoCo tasks, where the training curve of many several methods are still increasing while being terminated after 1M steps. It would be more convincing to run more env steps and compare algorithms when their returns get stable.
One minor thing: I would recommend rewriting the policy-parameter as or something else, since is often referred to some fixed or initial parameters.
Questions for authors
-
In Equation 17, it differentiates the value network with respect to its input to obtain first-order information. Are there any risks that this operation may cause numerical instability?
-
Table 6 shows that the correction scale varies in different environments. Can we find a way to automatically tune it?
3 comments
I would like to thank the authors for their responses and revisions, they resolved most of my concerns. Regarding the training duration, I recommend the authors to extend the horizon for all tasks in a future version, especially those in which the training curves are still climbing. Other than that, I have no further questions and still vote for acceptance.
Thank you for maintaining your positive assessment! Regarding extended training, the experiments take approximately 19 hours per benchmark on a single RTX 5090. We have been running these tests and expect to include the corresponding results in the final version.
We sincerely thank the reviewer for recognizing the importance of the problem, the clarity of our presentation, and the soundness of our theoretical analysis. We appreciate your constructive suggestions and address them below.
Thank you for pointing this out. We reran the experiments on Walker2D and Ant with the number of steps increased to 3×106. You can find our latest results in the revised manuscript.
Thank you for your suggestion. We have switched to Δθ.
We believe this question has two aspects. The first is whether the gradients themselves can explode numerically. The second is whether the gradients can accurately capture first-order changes in the value shift.
First, foundational studies from PINNs [1] have shown that gradient stiffness associated with PDE residuals typically depends on the properties of the equation itself. In particular, higher-order derivatives or large coefficients can substantially increase gradient magnitudes. The HJB equation, however, involves only first-order spatial derivatives. The coefficient of the derivative term is determined by the system dynamics, and the term itself remains modest in magnitude in continuous action spaces. The equation therefore avoids the additional gradient amplification associated with higher-order differentiation.
Second, whether the gradients can accurately capture first-order changes in the value shift ultimately depends on the accuracy of the critic’s estimates. Early in training, the critic has yet to converge sufficiently, so the estimated value shift may be considerably noisy. As training stabilizes, the critic’s estimates become more accurate, and the value-shift correction begins to take effect. This trend is consistent with the training behavior observed in Figure 5, particularly on Swimmer, where our method begins to yield sustained improvements in the middle to later stages of training.
We agree that automatically adapting α would be a valuable extension. One promising direction is to adjust the correction magnitude according to online estimates of the value-shift signal and its uncertainty, allowing stronger corrections when the estimate is reliable and more conservative corrections when its variance is high. These hypotheses involve methodological changes and require further experimental evidence for validation, so we leave them for future discussion.
Manuscript changes:
We hope these responses address your main concerns. Please kindly consider changing the rating if our responses have addressed your concerns.
References:
[1] Wang, S., Teng, Y., and Perdikaris, P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SISC. 2021.