acceptodds
Under review as a conference paper at ICLR 2027

Fisher-Trace State Sensitivity for Proximal Policy Optimization

Abstract

Deep reinforcement learning with neural critics often suffers from unstable value estimates and overly aggressive policy updates. Motivated by the intuition that states where the critic is highly sensitive deserve more cautious updates, we derive a state-level sensitivity measure for Proximal Policy Optimization (PPO) from the Fisher-information trace of the critic. We prove that, under the training conditions of standard PPO, this trace reduces to the squared norm of the value gradient with respect to states, and batch-wise standardization removes the unknown global scale. The resulting algorithm, GFIM-PPO, rescales the PPO surrogate loss with a bounded coefficient computed from this measure. On a diverse suite of reinforcement learning benchmarks, GFIM-PPO improves over PPO on the majority of tasks. Our analysis further confirms that it is the state-dependent construction of the weighting coefficient, rather than weighting alone, that drives the gains. Code is available at https://anonymous.4open.science/r/CAAC-F11F.

Primary area:
reinforcement learning
Keywords:
Reinforcement, Proximal Policy Optimization, Fisher-information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2nsTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GFIM-PPO, a variant of Proximal Policy Optimization that uses critic state sensitivity to adjust policy updates. The authors derive a sensitivity measure from the Fisher information trace of a Gaussian critic model, which reduces to the squared norm of the critic's state gradient under a homoscedastic assumption. This measure is used to down-weight policy updates in high-sensitivity states and adjust entropy regularization. Experiments on 14 reinforcement learning benchmarks show improvements over the PPO baseline on most tasks (questionable).

Critical strengths

The paper is well-organized and easy to follow. The proposed method is simple to implement and introduces only modest computational overhead. The authors also provide extensive ablation studies to examine the effects of different components and hyperparameter choices.

Critical weaknesses

Weak PPO baseline. The PPO baseline performs poorly on several MuJoCo tasks (e.g., Walker2d and Hopper) compared with publicly reported results from RL Baselines3 Zoo. This is a serious problem since it shows that both the baseline PPO and the proposed method fail to learn on the tasks. This also directly affects the effectiveness of ablation study, since similar phenomenon also shows in the appendix.

EnvironmentPaper PPOPaper GFIM-PPOZoo PPOZoo Training Steps
Hopper404.5417.12,410.41M
Walker2d23.2476.83,478.81M
HalfCheetah681.92,108.65,819.11M

The performance scores on MuJoCo (at least for these three tasks) (official Gymnasium documentation) environments are generally comparable between v3 and v4. For example, the PPO baseline in this paper achieves only 404.5 on Hopper, while RL Baselines3 Zoo reports a score of 2410 with the same 1M training steps. Although Zoo's results may benefit from hyperparameter tuning and selecting the best-performing runs, such a large performance gap still raises serious concerns about the quality of the PPO baseline and the fairness of the experimental comparisons.

EnvironmentPaper PPOPaper GFIM-PPOZoo PPOPaper StepsZoo Steps
MountainCarContinuous-v0-0.97-0.9088.3480k20k
Pendulum-v1-389.9-658.6-172.280k100k
BipedalWalker-v3-99.2-72.7287.980k5M
Acrobot-v1-107.2-153.4-73.580k1M
LunarLanderContinuous-70.9 (v3)-68.9 (v3)270.9 (v2)80k1M

For the experiments results above, algorithms just fail to learn. Authors should explain why some of them only get 80k step budgets.

Insufficient justification of the full method. In Table 2, GFIM-onlyent achieves higher mean returns than the full method on 4 out of 5 tasks, raising questions about the necessity of directed sample weighting. Although the authors provide extensive ablation studies, the results do not clearly establish whether the improvements come from the proposed weighting mechanism or could be largely explained by a simpler entropy adjustment.

Questions for authors

Given that the PPO baseline performs substantially worse than publicly reported results on a lot of tasks, could the authors explain the phenomenon? In particular, would the reported improvements still hold under a stronger PPO implementation with comparable training budgets?

In Table 2, GFIM-onlyent outperforms the full GFIM-PPO on 4 out of 5 tasks. The authors argue that the full method is more consistent across 14 tasks, but Table 1 does not include GFIM-onlyent. Could the authors provide a direct comparison across all 14 tasks to justify the necessity of the full method?

Prior works have explored critic-informed policy reweighting (Byun & Perrault, 2022) and the role of critic smoothness in policy optimization and state-aware exploration (Jesson et al., 2024). Could the authors clarify the main methodological novelty of GFIM-PPO beyond using the squared critic state-gradient norm as a weighting signal? In particular, what additional theoretical or algorithmic insights does the Fisher information formulation provide beyond a standard gradient-based sensitivity heuristic?

Reviewer_pufROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

GFIM-PPO uses the squared norm of the critic's state gradient to construct sample weights for the PPO surrogate and entropy terms. The paper interprets this signal through the Fisher trace of a Gaussian return model, evaluates the method on 14 control tasks, and reports component ablations and computational overhead.

Critical strengths

The method is simple to implement and requires no additional networks. The experiments use six seeds, report component ablations and runtime overhead, and show improvements over the PPO baseline on several MuJoCo tasks.

Critical weaknesses
  1. The central down-weighting principle lacks sufficient justification and direct validation. The Fisher derivation explains the squared state-gradient norm, but does not establish a connection between high sensitivity and unreliable estimates. The paper also does not directly measure the corresponding value or advantage estimation errors, or how these change after down-weighting. The return improvements therefore do not establish that the proposed noise-amplification mechanism has been mitigated.

  2. The additional value of the complete algorithm is unclear. In Table 2, GFIM-onlyent achieves higher mean returns than the complete method on four of five tasks. In the continuous-control implementation, entropy weighting amounts to adjusting an effective global entropy coefficient, yet a PPO control with a matched coefficient schedule is missing. The available results also do not sufficiently support the claim that the complete combination is more consistent than the ablated variants.

  3. The experimental scope and baseline validation are limited. Evaluation focuses on conventional state-input tasks and small networks, leaving effectiveness and cost on more complex robot-learning tasks or large-scale parallel PPO untested. Sharing hyperparameters helps control experimental differences, but baseline tuning and selection are insufficiently documented. The low PPO returns on some tasks also leave it unclear how much the gains depend on the baseline configuration.

Questions for authors
  1. Can you provide direct evidence relating sensitivity to estimation error, together with diagnostics before and after down-weighting?

  2. Under equal tuning budgets, do consistent gains remain over PPO with a matched entropy-coefficient schedule and over GFIM-onlyent?

  3. Please specify PPO-TD's weighting formula, normalization, clipping, and treatment of the entropy term.

  4. Please extend the evaluation to Isaac Lab or an equivalent large-scale parallel robot-learning benchmark, comparing against reasonably tuned PPO and GFIM-onlyent. Report the number of parallel environments, total rollout size, returns, and wall-clock/memory overhead to assess applicability and scaling beyond the current experimental scope.

Reviewer_4jFnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GFIM-PPO, which uses the Fisher-information trace of a Gaussian critic to measure state sensitivity. Under a homoscedastic assumption, this measure reduces to the squared norm of the critic’s state gradient. The resulting sensitivity is used to reweight PPO policy updates and entropy regularization. Experiments across 14 reinforcement learning tasks demonstrate improvements over the PPO baseline.

Critical strengths
  • The idea of using critic state sensitivity to adapt policy updates is intuitive and practically meaningful.

  • The method improves performance on 12 of 14 benchmark tasks with relatively modest computational overhead. The extensive ablations provide useful insights into the weighting and entropy mechanisms.

Critical weaknesses
  • Limited methodological novelty and insufficient justification of the sensitivity measure. Under the homoscedastic Gaussian assumption, the proposed Fisher-information trace reduces to the squared norm of the critic’s state gradient, which is a well-established sensitivity measure. Related ideas have been explored in value-gradient-based learning [1] and critic-reliability-based policy weighting [2]. More importantly, a large value gradient does not necessarily indicate inaccurate value predictions or unreliable advantage estimates. An accurately learned value function may naturally exhibit steep gradients. The paper lacks direct evidence establishing the relationship between the proposed sensitivity measure and actual critic or advantage estimation errors.

[1] Voelcker et al., VALUE GRADIENT WEIGHTED MODEL-BASED REINFORCEMENT LEARNING, ICLR 2022

[2] Byun et al., Normality-Guided Distributional Reinforcement Learning for Continuous Control, TMLR 2025

  1. Insufficient comparisons with strong PPO baselines. The experiments only compare GFIM-PPO against vanilla PPO and PPO-TD, without considering representative PPO improvements such as Phasic Policy Gradient (Cobbe et al., ICML 2021) and Truly PPO (Wang et al., UAI 2019). More importantly, the PPO baseline exhibits surprisingly weak performance on several MuJoCo tasks. Given the well-documented sensitivity of PPO to implementation details and hyperparameter choices, comparisons against well-tuned, established PPO implementations are necessary to establish the actual effectiveness of GFIM-PPO.
Questions for authors
  • Can the authors demonstrate that the proposed Fisher-based sensitivity correlates with actual critic prediction errors or advantage estimation errors, rather than merely reflecting steep value-function landscapes?

  • Can the authors provide comparisons against established, well-tuned PPO implementations (e.g., CleanRL and Stable-Baselines3), as well as more representative PPO variants, such as PPGPhasic Policy Gradient (ICML 2021) and Truly PPO (UAI 2019)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.