- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that feature rank is a poor diagnostic for representation collapse in PPO. It analyzes cross-state gradient coupling through the policy heads, relates its feature component to the feature spectrum and norms, and proposes the participation ratio of centered, normalized policy outputs as an alternative diagnostic. Building on this, the paper proposes FF-PPO, which regularizes feature drift using a penalty based on a Fisher pullback. Experiments on six MuJoCo tasks, together with additional stress tests, provide evidence of reduced feature coupling and improved robustness in selected settings.
Critical strengths
- The critique of rank-based diagnostics is clear and convincing, and the proposed diagnostic measures offer a new perspective on diagnosing and improving PPO training.
Critical weaknesses
-
Detach KL in the main experiment and the Jacobian-corrected metric are both more faithful to the quantity the paper aim to measure, but both underperform FF-PPO-log. The paper presents dropping the Jacobian only as a deliberate approximation and does not explain why the more exact objects underperform. It is therefore unclear whether the gain comes from the Fisher metric or from the approximation itself.
-
Equation (7) includes both the feature factor and the Fisher weight, which is constant when the policy standard deviation is state-independent. This holds in the contextual bandit experiment but generally not in the MuJoCo experiments. Reporting the full Fisher-weighted coupling on MuJoCo would provide clearer evidence for the proposed mechanism.
Questions for authors
What explains the performance gap between FF-PPO-log and the detached KL and Jacobian-corrected variant?
Does the full coupling in Eq. (7) show the same relationship with trust-region violations and returns in the MuJoCo experiments?