- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GFIM-PPO, a variant of Proximal Policy Optimization that uses critic state sensitivity to adjust policy updates. The authors derive a sensitivity measure from the Fisher information trace of a Gaussian critic model, which reduces to the squared norm of the critic's state gradient under a homoscedastic assumption. This measure is used to down-weight policy updates in high-sensitivity states and adjust entropy regularization. Experiments on 14 reinforcement learning benchmarks show improvements over the PPO baseline on most tasks (questionable).
Critical strengths
The paper is well-organized and easy to follow. The proposed method is simple to implement and introduces only modest computational overhead. The authors also provide extensive ablation studies to examine the effects of different components and hyperparameter choices.
Critical weaknesses
Weak PPO baseline. The PPO baseline performs poorly on several MuJoCo tasks (e.g., Walker2d and Hopper) compared with publicly reported results from RL Baselines3 Zoo. This is a serious problem since it shows that both the baseline PPO and the proposed method fail to learn on the tasks. This also directly affects the effectiveness of ablation study, since similar phenomenon also shows in the appendix.
| Environment | Paper PPO | Paper GFIM-PPO | Zoo PPO | Zoo Training Steps |
|---|---|---|---|---|
| Hopper | 404.5 | 417.1 | 2,410.4 | 1M |
| Walker2d | 23.2 | 476.8 | 3,478.8 | 1M |
| HalfCheetah | 681.9 | 2,108.6 | 5,819.1 | 1M |
The performance scores on MuJoCo (at least for these three tasks) (official Gymnasium documentation) environments are generally comparable between v3 and v4. For example, the PPO baseline in this paper achieves only 404.5 on Hopper, while RL Baselines3 Zoo reports a score of 2410 with the same 1M training steps. Although Zoo's results may benefit from hyperparameter tuning and selecting the best-performing runs, such a large performance gap still raises serious concerns about the quality of the PPO baseline and the fairness of the experimental comparisons.
| Environment | Paper PPO | Paper GFIM-PPO | Zoo PPO | Paper Steps | Zoo Steps |
|---|---|---|---|---|---|
| MountainCarContinuous-v0 | -0.97 | -0.90 | 88.34 | 80k | 20k |
| Pendulum-v1 | -389.9 | -658.6 | -172.2 | 80k | 100k |
| BipedalWalker-v3 | -99.2 | -72.7 | 287.9 | 80k | 5M |
| Acrobot-v1 | -107.2 | -153.4 | -73.5 | 80k | 1M |
| LunarLanderContinuous | -70.9 (v3) | -68.9 (v3) | 270.9 (v2) | 80k | 1M |
For the experiments results above, algorithms just fail to learn. Authors should explain why some of them only get 80k step budgets.
Insufficient justification of the full method. In Table 2, GFIM-onlyent achieves higher mean returns than the full method on 4 out of 5 tasks, raising questions about the necessity of directed sample weighting. Although the authors provide extensive ablation studies, the results do not clearly establish whether the improvements come from the proposed weighting mechanism or could be largely explained by a simpler entropy adjustment.
Questions for authors
Given that the PPO baseline performs substantially worse than publicly reported results on a lot of tasks, could the authors explain the phenomenon? In particular, would the reported improvements still hold under a stronger PPO implementation with comparable training budgets?
In Table 2, GFIM-onlyent outperforms the full GFIM-PPO on 4 out of 5 tasks. The authors argue that the full method is more consistent across 14 tasks, but Table 1 does not include GFIM-onlyent. Could the authors provide a direct comparison across all 14 tasks to justify the necessity of the full method?
Prior works have explored critic-informed policy reweighting (Byun & Perrault, 2022) and the role of critic smoothness in policy optimization and state-aware exploration (Jesson et al., 2024). Could the authors clarify the main methodological novelty of GFIM-PPO beyond using the squared critic state-gradient norm as a weighting signal? In particular, what additional theoretical or algorithmic insights does the Fisher information formulation provide beyond a standard gradient-based sensitivity heuristic?