- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper shows local credit assignment that reduces local gradient variance can result in the increment of the joint policy gradient. To address this problem, PLACID is proposed to select the corrections via the change of the joint variance.
Critical strengths
-
The counterexample is intuitive and informative to characterize the failure of variance guarantee.
-
PLACID is well-motivated and supported by a lot of experiments.
Critical weaknesses
- The relationship between the variance and the performance is my main concern. The theory does not provide value improvement guarantee. I know it might be hard, but more explanation and experiments on non-toy games would strengthen the claim of the paper.
Questions for authors
- Can you discuss the relationship between PLACID and other variance reduction tricks like advantage normalization and gradient clipping in PPO? Would PLACID's benefits persist when these tricks are combined? In other word, if the goal is to control the gradient, why should I use PLACID instead of all these standard tricks in practice?