acceptodds
Under review as a conference paper at ICLR 2027

Clipped Diagonal Natural Policy Gradient

Abstract

Natural policy gradient (NPG) methods account for the local geometry of the policy distribution, but computing directions preconditioned by the Fisher information matrix is expensive. Approximating this matrix with its diagonal yields a computationally efficient NPG algorithm, but such an algorithm has shown weak and sometimes unstable empirical performance in prior studies. We show that inverse-diagonal scaling can create nonlocal parameter displacements, while diagonal normalization can misestimate the update's length under the full Fisher metric. We introduce Clipped Diagonal NPG (CD-NPG). It clips each coordinate of the inverse-diagonal direction at a multiple of the median coordinate magnitude within its layer. It then uses the current minibatch to estimate the clipped direction's full-Fisher quadratic without forming the matrix. The method uses this estimate to rescale the direction to a target local policy change. These modifications preserve the asymptotic computational and memory complexities of diagonal NPG. Under matched hyperparameter-search budgets on six MuJoCo tasks, CD-NPG obtains returns comparable to or higher than policy gradients optimized with Adam or Muon on most tasks. Comparisons with two richer, more expensive NPG approximations are task- and training-setting-dependent. Diagnostics show substantially better local-KL calibration than diagonal normalization, while clipping ablations show that the clipped direction has fewer extreme finite-step departures from the local Fisher model than the unclipped direction.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, policy gradient, natural policy gradient

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pmdqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CD-NPG, which approximates the natural policy gradient direction using a diagonal approximation of the Fisher matrix. CD-NPG uses the full Fisher metric for step normalisation to avoid misestimating the update length under the diagonal approximation. To control extreme coordinate updates, CD-NPG clips the update using thresholds based on the median coordinate magnitude within each layer. Experiments on six MuJoCo benchmarks show that CD-NPG outperforms Adam-PG and Muon-PG under certain hyperparameter configurations.

Critical strengths
  1. Computationally practical design. The method addresses the computational bottleneck of natural policy gradient in practical training by eliminating the need to explicitly construct the full Fisher matrix or perform iterative linear solves.

  2. Clear problem analysis. The paper clearly distinguishes two issues arising from diagonal Fisher approximations during training: miscalibration of the update length and excessively large coordinate updates.

Critical weaknesses
  1. The proposed mechanism is insufficiently motivated and validated. The paper explains why extreme coordinate updates should be controlled, but the specific choice of median-based clipping is not sufficiently justified. Its benefits have not been isolated and validated against alternatives such as adaptive clipping based on the mean or RMS, or damping as a replacement for clipping. Consequently, it remains unclear whether median-based clipping is the best choice.

  2. The experimental evidence is not convincing.

    • The toy example in Figure 1 is misleading, The exact Fisher matrix in this example is already diagonal, so the raw proposal is the exact natural gradient direction computed from a noisy gradient. Therefore the example is not directly related to the diagonal approximation of sample-based Fisher that is the focus of the paper.

    • The experimental setup also raises concerns. Curves obtained by independently ranking different hyperparameter configurations do not provide a particularly fair comparison, especially because some search ranges appear unreasonable. For CG-NPG, the largest KL radius is 10−310^{-3}, whereas a commonly used setting of TRPO is typically 10−210^{-2}. For both CG-NPG and K-FAC-NPG, the search includes very small damping values (10−610^{-6}), which often directly cause training instability in practice. The learning-rate range for Muon is also relatively small.

    • The experiments lack training diagnostics beyond final returns. The KL diagnostics are limited in scope. There are no KL traces for the baselines under the main protocol and no learning curves, so it remains unclear whether CD-NPG actually controls policy change better than the baselines during training.

    • Given the method’s computational advantages, the evaluation at larger scales is insufficient. The Atari experiments are limited to four environments in the appendix, and the relatively weak performance is not adequately explained.

Questions for authors
  1. Could the authors explain why median-based clipping is more appropriate than alternatives? Even without additional experimental comparisons, could the authors provide a stronger justification for this choice?

  2. Could the authors clarify the role of clipping in Figure 1, or give an example where the exact Fisher is not diagonal? At a minimum, the population NPG direction should be removed from the figure to avoid misleading readers.

  3. I understand that appropriate hyperparameter choices depend on implementation details. Could the authors provide more comprehensive hyperparameter-search results? At a minimum, could the authors explain how the current search ranges were determined?

  4. Does CD-NPG achieve more stable KL control during training? Or could the authors clarify the source of its performance advantages?

Reviewer_7LsJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts from the following observation. Diagonal NPG, the cheapest nontrivial natural-gradient approximation, which replaces F^{-1}g by diag(F)^{-1}g and therefore costs about as much as Adam, has been reported by Gebotys et al. (2022) and Sun (2026) to perform poorly and sometimes unstably; the paper takes the reliability of this finding as its starting point. Within this framing the authors diagnose two distinct failure modes. First, small diagonal Fisher entries amplify gradient noise into large single-coordinate parameter displacements, so the update leaves the region where the local Fisher model is trustworthy (direction failure). Second, the diagonal quadratic q^T D q drops the cross terms of q^T F q and systematically mismeasures the full-Fisher step length, with an error factor that grows linearly in the parameter count (Eq. 8).

The proposed method, CD-NPG, applies two fixes. A coordinatewise clip at c_rel times the within-layer median of the proposed direction addresses the direction failure. A matrix-free scalar quadratic form along the clipped direction, computed with one JVP and an output-space Fisher contraction, rescales the step to a target KL radius and addresses the step-length failure. Both fixes preserve the asymptotic compute and memory cost of diagonal NPG and Adam.

Validation, as reported by the authors, covers six MuJoCo tasks under on-policy and off-policy (PPO-style) training with a matched 6x6 hyperparameter grid and ten seeds per configuration. The authors report that CD-NPG matches or exceeds Adam+PG and Muon+PG on most tasks, with a clearer margin off-policy, while comparisons against CG-NPG and K-FAC-NPG are task-dependent. Appendix extensions cover batch size 256, a wider actor, cost and memory benchmarks, and four Atari tasks (on-policy only, where K-FAC-NPG is strongest). Mechanism diagnostics (Figures 8-12) show that full-Fisher normalization tracks the target KL while diagonal normalization persistently overshoots, with an error that grows with parameter count, and that clipping mainly suppresses the extreme finite-step KL tail associated with large coordinate steps.

Critical strengths
  1. The failure diagnosis is rigorous and honest. Each failure mode is exhibited cleanly: the correlated-Fisher example in Eq. 8 for the step-length error, and the two-dimensional bandit in Appendix E for the direction error, where the unclipped update has an exact KL 153 times its predicted value while the clipped update lands at 1.35 times. Proposition 1 and Eq. 20 delimit the value of clipping to estimation-error and finite-step regimes, and the authors do not invent theory to justify a heuristic. The mechanism diagnostics in Figures 8-12 are the most convincing part of the paper.

  2. The cost argument is credible and thoroughly evidenced. The theoretical FLOP ratio over Adam is 7/3 (about 5/3 including a matched critic), the measured actor update takes 1.28 to 1.43 times Adam across tested widths, persistent memory matches Adam and lies between CG-NPG and K-FAC-NPG, while peak memory stays close to CG-NPG and substantially below K-FAC-NPG (Table 3).

Critical weaknesses
  1. The novelty is a recombination of known components, and the theory provides no support for it. Coordinatewise clipping after diagonal preconditioning has its closest algebraic precedent in Sophia (Liu et al., 2024), with the differences the authors note (policy Fisher versus loss Hessian, layer-relative median threshold versus fixed cap), and Fisher-quadratic step normalization follows TRPO (Schulman et al., 2015), as acknowledged in Appendix A. The median-relative within-layer threshold is an incremental change. Moreover, the single proposition in the paper does not model the actual algorithm (no EMA, no estimation error, no clipping rule), the authors state it gives no universal ranking of practical optimizers, and Eq. 20 shows clipping cannot improve the ideal local gain even when F equals D. What remains is an engineering combination whose justification must come entirely from the experiments, and the experiments have their own problems (Weaknesses 2 and 3). For an ICLR-level algorithmic contribution, this is the central concern.

  2. There is no held-out evaluation: the same ten tuning seeds are used both to select the winning configuration per task and method and to report its return. Different methods have tuning landscapes with different noise levels, so the optimism of winner selection differs across methods, and the headline claim of matching or exceeding Adam and Muon on most tasks is therefore a biased comparison. The setting-rank curves mitigate this only partially, since they are summaries over the same seeds and no paired significance test is reported.

  3. The key clipping ablation (Figure 13) has an asymmetric selection bias. At each target radius delta, the clipped curve shows the best of six c_rel settings, while the unclipped curve is a single configuration with c_rel equal to infinity. Selecting the best of six on the same tuning seeds directly inflates the clipped curve. The claim that clipping widens the range of usable target radii is therefore confounded with selection bias and cannot be attributed to clipping itself.

  4. Baseline coverage and scope have gaps. TRPO with its backtracking line search is not compared; the implemented CG-NPG stops at the locally normalized step and omits TRPO's line search (Appendix D.3), and ACKTR is cited but not implemented. The primary MuJoCo evaluation uses fixed-variance Gaussian MLPs of width 64, with a (256,256) actor repeat and a four-task Atari study in the appendices, but learned variances, recurrent policies, and larger scales remain untested. On Atari, K-FAC-NPG dominates across most ranks and tasks; the paper's explanation (within-layer structure being particularly useful for convolutional policies) is consistent but speculative and rests on four on-policy tasks.

Questions for authors
  1. Can you rerun the per-task winning configurations on held-out seeds that were not used for tuning, and report paired significance tests?

  2. Can the delta-matched comparison of Figure 13 be reproduced under a symmetric aggregation rule on both sides, for example best-of-grid versus best-of-grid, or full rank curves for both?

  3. The scalar Q_t is estimated from only 64 states, and Figure 12 shows a large shift between training states and held-out states. In practice, does the noise of Q_t occasionally shrink steps too much or let them overshoot, and would an EMA smoother or a clamp on Q_t help?

Reviewer_u8CYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Clipped Diagonal Natural Policy Gradient (CD-NPG) addressing two limitations of diagonal NPG: extreme coordinate displacements from inverse-diagonal preconditioning and inaccurate step length estimates from diagonal Fisher normalization. The method clips the preconditioned direction relative to the median coordinate magnitude within each layer, then normalizes it using a matrix-free evaluation of its full-Fisher quadratic. Analytic examples and empirical diagnostics motivate the two modifications. Experiments on six MuJoCo tasks and four Atari games compare returns, hyperparameter sensitivity, and computational cost against Adam, Muon, CG-NPG, and K-FAC-NPG, positioning CD-NPG as a competitive alternative with modest actor-update overhead over Adam.

Critical strengths
  1. The paper separates extreme coordinate displacements from inaccurate step-length calibration, using analytic examples to motivate a targeted modification for each. Comparisons of clipped and unclipped directions, diagonal and full-Fisher normalization, and finite-step KL on training and held-out states provide evidence for how the modifications affect policy updates.

  2. The study compares multiple optimizer families across MuJoCo and Atari, reports performance across hyperparameter grids, and examines sensitivity to actor width and minibatch size. The paper details runtime and memory costs and clearly acknowledges that the method does not recover full NPG geometry or guarantee finite-step KL control.

Critical weaknesses
  1. The main contribution is a practical combination of established ideas: diagonal Fisher preconditioning, coordinate clipping, and Fisher-based step normalization. The paper provides useful justification and empirical support for this combination, but the individual components have clear precedents, and the theoretical analysis does not directly characterize the complete practical algorithm.

  2. The continuous-control experiments use fixed-variance Gaussian policies, leaving uncertainty about whether the findings extend to policies with learned variances. In the Atari experiments, K-FAC-NPG performs strongest overall. These results support CD-NPG as a useful practical alternative, while its performance advantages remain dependent on the setting.

Questions for authors
  1. The CG-NPG and K-FAC-NPG baselines use damping, whereas CD-NPG uses coordinate clipping with a small numerical floor on the Fisher diagonal. Since damping can also mitigate large updates in weak-curvature directions, how should we interpret the difference between these mechanisms in addressing the instability discussed in Section 3.3?

  2. How is the Fisher diagonal computed for the convolutional layers in the Atari experiments? Appendix C.2 explains why Equation 21 does not directly apply to shared convolutional weights. Does the implementation retain the cross-position terms exactly, or use an additional approximation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.