- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes CD-NPG, which approximates the natural policy gradient direction using a diagonal approximation of the Fisher matrix. CD-NPG uses the full Fisher metric for step normalisation to avoid misestimating the update length under the diagonal approximation. To control extreme coordinate updates, CD-NPG clips the update using thresholds based on the median coordinate magnitude within each layer. Experiments on six MuJoCo benchmarks show that CD-NPG outperforms Adam-PG and Muon-PG under certain hyperparameter configurations.
Critical strengths
-
Computationally practical design. The method addresses the computational bottleneck of natural policy gradient in practical training by eliminating the need to explicitly construct the full Fisher matrix or perform iterative linear solves.
-
Clear problem analysis. The paper clearly distinguishes two issues arising from diagonal Fisher approximations during training: miscalibration of the update length and excessively large coordinate updates.
Critical weaknesses
-
The proposed mechanism is insufficiently motivated and validated. The paper explains why extreme coordinate updates should be controlled, but the specific choice of median-based clipping is not sufficiently justified. Its benefits have not been isolated and validated against alternatives such as adaptive clipping based on the mean or RMS, or damping as a replacement for clipping. Consequently, it remains unclear whether median-based clipping is the best choice.
-
The experimental evidence is not convincing.
-
The toy example in Figure 1 is misleading, The exact Fisher matrix in this example is already diagonal, so the raw proposal is the exact natural gradient direction computed from a noisy gradient. Therefore the example is not directly related to the diagonal approximation of sample-based Fisher that is the focus of the paper.
-
The experimental setup also raises concerns. Curves obtained by independently ranking different hyperparameter configurations do not provide a particularly fair comparison, especially because some search ranges appear unreasonable. For CG-NPG, the largest KL radius is , whereas a commonly used setting of TRPO is typically . For both CG-NPG and K-FAC-NPG, the search includes very small damping values (), which often directly cause training instability in practice. The learning-rate range for Muon is also relatively small.
-
The experiments lack training diagnostics beyond final returns. The KL diagnostics are limited in scope. There are no KL traces for the baselines under the main protocol and no learning curves, so it remains unclear whether CD-NPG actually controls policy change better than the baselines during training.
-
Given the method’s computational advantages, the evaluation at larger scales is insufficient. The Atari experiments are limited to four environments in the appendix, and the relatively weak performance is not adequately explained.
-
Questions for authors
-
Could the authors explain why median-based clipping is more appropriate than alternatives? Even without additional experimental comparisons, could the authors provide a stronger justification for this choice?
-
Could the authors clarify the role of clipping in Figure 1, or give an example where the exact Fisher is not diagonal? At a minimum, the population NPG direction should be removed from the figure to avoid misleading readers.
-
I understand that appropriate hyperparameter choices depend on implementation details. Could the authors provide more comprehensive hyperparameter-search results? At a minimum, could the authors explain how the current search ranges were determined?
-
Does CD-NPG achieve more stable KL control during training? Or could the authors clarify the source of its performance advantages?