acceptodds
Under review as a conference paper at ICLR 2027

Fisher-Feature Proximal Policy Optimization

Abstract

Proximal Policy Optimization (PPO) loses plasticity in its trunk, and the usual diagnostic is the rank of the penultimate feature matrix. Proximal Feature Optimization (PFO) follows that diagnostic by penalizing feature drift equally in every direction. We argue that the diagnostic is the weak link. The policy sees the features only through two linear heads whose width is set by the action space, so rank, computed without the heads, cannot separate representations the policy treats very differently; and rank is a count settled at the tail of the spectrum, so it misses how variance is shared. We derive how an update on one state leaks into the clipped ratio of another: the leakage is governed by the cross-state coupling of the features, their squared inner products, which depends on how the spectrum is spread and on the feature norm, and not on a count. We measure the angular part by the participation ratio of the projected outputs, essentially the inverse mean squared cosine between states, and replace the isotropic penalty by the unique quadratic penalty that measures policy change, the Fisher information pulled back through the heads (FF-PPO). On six MuJoCo tasks the predicted mechanism is visible directly: FF-PPO lowers the coupling by more than an order of magnitude relative to PPO and several-fold relative to PFO, and wherever the penalties help, the methods order by coupling exactly as they order by trust-region excess. The gain in return grows with the width of the policy-visible subspace, where FF-PPO separates from PFO, and with the strength of the non-stationarity: with more optimization epochs per batch or under changing dynamics, PPO collapses while FF-PPO keeps learning.

Primary area:
reinforcement learning
Keywords:
Proximal Policy Optimization, Proximal Feature Optimization, Loss of Placticity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RYQPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that feature rank is a poor diagnostic for representation collapse in PPO. It analyzes cross-state gradient coupling through the policy heads, relates its feature component to the feature spectrum and norms, and proposes the participation ratio of centered, normalized policy outputs as an alternative diagnostic. Building on this, the paper proposes FF-PPO, which regularizes feature drift using a penalty based on a Fisher pullback. Experiments on six MuJoCo tasks, together with additional stress tests, provide evidence of reduced feature coupling and improved robustness in selected settings.

Critical strengths
  1. The critique of rank-based diagnostics is clear and convincing, and the proposed diagnostic measures offer a new perspective on diagnosing and improving PPO training.
Critical weaknesses
  1. Detach KL in the main experiment and the Jacobian-corrected metric are both more faithful to the quantity the paper aim to measure, but both underperform FF-PPO-log. The paper presents dropping the Jacobian only as a deliberate approximation and does not explain why the more exact objects underperform. It is therefore unclear whether the gain comes from the Fisher metric or from the approximation itself.

  2. Equation (7) includes both the feature factor and the Fisher weight, which is constant when the policy standard deviation is state-independent. This holds in the contextual bandit experiment but generally not in the MuJoCo experiments. Reporting the full Fisher-weighted coupling on MuJoCo would provide clearer evidence for the proposed mechanism.

Questions for authors

What explains the performance gap between FF-PPO-log and the detached KL and Jacobian-corrected variant?

Does the full coupling in Eq. (7) show the same relationship with trust-region violations and returns in the MuJoCo experiments?

Reviewer_kKAFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why PPO can become less effective during training and argues that existing ways of measuring this problem are insufficient. The paper explain how changes in the learned features can interfere with the learning process and propose a modification of PPO, called FF-PPO, that limits these changes based on how much they affect the policy. The main tool is to use the policy's Fisher information.

Critical strengths
  • The method improves on existing approaches by giving more importance to feature changes that affect the policy. It can also be incorporated into PPO without major modifications.
  • The paper provides a simple experiment in Section 3.6 showing that learning can slow down significantly even when feature rank stays unchanged. This helps explain why rank alone is not a reliable measure of learning performance.
Critical weaknesses

Major:

  • It seems to me that the paper emphasizes mathematical motivation quite prominently from the beginning. However, I find the theoretical contribution quite limited. The main results rely on standard identities such as Proposition 1. Also, the derivations in Appendix D regarding Gaussian FI aren't new. If I understand correctly, the paper doesn't provide actual theoretical guarantees showing that the method prevents representation collapse or improves learning stability.

  • Furthermore, I believe there is a gap between the mathematical motivation and the penalty that is actually implemented. The paper motivates FF-PPO using the FI metric and argues that it penalizes feature changes according to their effect on the policy. To keep your notation from Proposition 1, if we expand the KL for a small perturbation δϕ,\delta \phi, we get KL(πy∥πy+δy)=12δϕ⊤W⊤FWδϕ+o(∥δϕ∥2),\mathrm{KL}(\pi_y\|\pi_{y+\delta y})=\frac12\delta\phi^\top W^\top FW\delta\phi+o(\|\delta\phi\|^2), where the policy outputs are y=Wϕy=W\phi and WW contains the weights of the policy's mean and scale heads.

On the other hand, equation (14) poses the penalty Es[δz(s)⊤W⊤F(s)Wδz(s)].\mathbb E_s\left[\delta z(s)^\top W^\top F(s)W\delta z(s)\right].

Due to ϕ=f(z),\phi=f(z), equation (14) misses the Jacobian of ff in the expression of the M.M. The paper argues that the Jacobian is dropped but I don't see the justification for that.

Minor:

  • I found the abstract quite dense. A simpler writing without much technical jargon would help the reader understand the main contribution of the paper before actually starting to read it.
  • Equation (1) appears quite suddenly without enough presentation of the setup and H\mathcal{H} and P\mathcal{P} are not properly defined.
  • I manually checked the references one by one. They seem correct to me apart from "Andrei Lascu et al. Fisher-rao proximal policy optimization. arXiv preprint, 2025." The author's name is incomplete and the paper's title is incorrect. Please, double-check this reference.
Questions for authors
  • Can the authors explain the technical novelty of the paper?
  • Can the authors explain the mathematical gap raised above?
Reviewer_Auq5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies performance degradation in PPO under repeated optimization and analyzes its mechanism through cross-state update interference. It proposes FF-PPO, which constrains feature drift using a Fisher-weighted regularizer. Experiments on MuJoCo continuous-control tasks evaluate its effects on training stability and final return.

Critical strengths
  1. The experiments provide some evidence of the method’s effectiveness. On certain tasks and under increased numbers of optimization epochs, FF-PPO achieves better returns or greater stability than PPO and PFO. The improvement on Humanoid is particularly noticeable. Evaluation across multiple random seeds also strengthens the credibility of these results.
Critical weaknesses
  1. The connection between the proposed method and its motivation is not sufficiently established. The training degradation discussed in the paper is closely related to the distribution shift between the behavior policy used to collect data and the policy currently being optimized. The proposed method primarily mitigates the effects of this shift by constraining feature changes. However, it remains unclear how it distinguishes harmful update interference from changes necessary for policy improvement. Reducing policy drift alone is therefore insufficient to establish that the method preserves the ability to continue learning.

  2. The appropriate degree of update conservatism is insufficiently justified. The current policy’s sensitivity to perturbations is not equivalent to the actual discrepancy between the behavior and current policies. In particular, when the policy has not yet developed clear action preferences and the two policies remain close, allowing more substantial updates may help the policy develop useful preferences. The paper lacks a direct analysis of this situation. Consequently, it remains unclear whether the proposed constraint could impede necessary policy improvement, or why it should strike an appropriate balance between stability and update magnitude.

  3. The experimental results also reveal this limitation. On Swimmer, FF-PPO reduces coupling, yet its return decreases from approximately 112 for PPO to approximately 51. This indicates that reducing update interference does not necessarily improve learning, and that excessive constraints may instead limit adaptation. Although the authors discuss this failure case, it remains unresolved, limiting how strongly the empirical evidence supports the paper’s motivation.

Questions for authors
  1. When the behavior and current policies are close, but the current policy lacks clear action preferences, could the method restrict beneficial updates? Could the authors investigate this by independently controlling policy concentration and the discrepancy between the two policies?
  2. How does the proposed regularizer determine the appropriate constraint strength based on the actual discrepancy between the behavior and current policies? What additional justification does Fisher sensitivity provide?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.