- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the loss of action information caused by limited floating-point precision when PPO employs a tanh-squashed Gaussian policy. The authors show that once actions saturate at the boundaries, the original sampled pre-tanh latents cannot be accurately recovered, leading to biases in likelihood calculations and policy gradients during policy updates. The paper theoretically analyzes this issue and compares two solutions: retaining the original latent variables (DUAL) and computing probabilities over quantization intervals (CELL). Experiments on three MuJoCo simulation environments—HalfCheetah, Walker2d, and Ant—as well as with an independent PPO implementation demonstrate that these methods substantially reduce extreme approximate KL values and numerical instability, but do not consistently improve task returns.
Critical strengths
- In-depth problem analysis with clear theoretical derivations. The paper identifies the loss of action information caused by limited floating-point precision in tanh-squashed PPO and mathematically demonstrates that policy gradient bias can arise even when the initial likelihood ratio is 1. This analysis provides valuable insights into potential numerical instabilities in PPO training and offers theoretical contributions.
- Controlled experimental validation of numerical stability. The authors conduct comparative experiments across three MuJoCo environments and further verify the existence of the problem using an independent PPO implementation. The results show that retaining the original latent variables or using CELL likelihood can significantly reduce extreme KL divergence values and numerical anomalies, supporting the paper's proposed mechanism analysis.
Critical weaknesses
-
Limited generalizability of the research problem and methodological novelty. This paper primarily investigates a non-default tanh-squashed gSDE PPO configuration in SB3, using a relatively large initial exploration scale (log std = 0), which causes many actions to saturate at the boundaries. The appendix experiments show that simply adjusting the exploration initialization to -2 can significantly improve the training stability and task returns of RECON. Furthermore, the validation using an independent implementation relies on artificially introducing mean shifts to induce action saturation. Therefore, the paper does not sufficiently demonstrate that this issue has a widespread impact on standard PPO configurations or real-world robotic reinforcement learning. Additionally, both DUAL and CELL are based on existing techniques, making the methodological novelty relatively limited.
-
Insufficient validation of improvements in practical control performance. Although the paper demonstrates improved numerical stability across three MuJoCo simulation environments, the main benefits are reflected in numerical metrics such as peak approximate KL divergence, without sufficient evidence that these improvements consistently enhance actual policy performance. For example, in the Ant environment, both proposed methods achieve lower task returns than the original RECON baseline, and the best-performing method varies across tasks. Moreover, the experiments are limited to three locomotion environments with only five random seeds per setting, lacking evaluations on more complex robotic manipulation tasks or real-world robots. These limitations weaken the persuasiveness of the paper's conclusions regarding practical robotic reinforcement learning applications.
Questions for authors
-
Regarding the generalizability of the problem: This paper primarily investigates tanh-squashed gSDE PPO with a relatively large initial exploration scale (log std = 0). Given that standard PPO does not use this configuration by default, could the authors further clarify how frequently and severely this numerical issue occurs in other commonly used squashed-policy configurations or practical robotic reinforcement learning settings?
-
Regarding the impact of hyperparameter settings: Table 3 shows that simply adjusting the initial exploration scale from 0 to -2 can significantly improve the training stability and task returns of RECON. Could the authors further compare RECON-INIT2, DUAL, and CELL under consistent experimental settings to demonstrate that the proposed methods offer clear advantages over reasonable hyperparameter tuning?
-
Regarding practical task performance: Although DUAL and CELL significantly reduce peak approximate KL divergence, task returns do not show consistent improvements. In particular, both methods achieve lower returns than RECON in the Ant environment. Could the authors explain this phenomenon and further clarify whether improvements in numerical stability can translate into reliable gains in policy performance?
-
Regarding the appropriateness of evaluation metrics: The paper primarily uses peak approximate KL divergence to measure training stability. However, the authors also acknowledge that extremely large approximate KL values do not necessarily accurately reflect the true KL divergence between policy distributions. Could the authors provide additional training stability metrics, such as actual changes in policy distributions or more detailed statistics on gradient anomalies, to further validate the effectiveness of the proposed methods?