acceptodds
Under review as a conference paper at ICLR 2027

The Stored Action Does Not Identify the Sampled Latent: Finite-Precision Likelihood Replay in Squashed Policies

Abstract

A stored action is not always a faithful record of the sample that produced it. In a tanh-squashed policy, several large Gaussian samples can become the same float32 action, exactly or . Applying the inverse transform cannot recover the original sample. We study a Stable-Baselines3 PPO pathway that nevertheless reconstructs this value when evaluating action likelihoods. Its likelihood ratio initially equals one because collection and optimisation use the same reconstruction, but its gradient already differs from the gradient at the sampled value. Direct logging detects boundary collisions within the first eight environment interactions. Across three MuJoCo tasks, storing the sampled pre-tanh value substantially reduces peak approximate KL despite equal or greater boundary incidence. On HalfCheetah, the median falls from to . A separate PPO implementation reproduces the difference under a shared saturation stressor. We also show how a measured transform threshold predicts boundary exposure and how exploration initialisation changes that exposure. The practical distinction is between avoiding information loss and preserving the information needed for replay. Store the sampled latent when later updates evaluate its likelihood. When only a rounded action remains, model the probability of its rounding cell rather than treating a clamped inverse as the original sample.

Primary area:
reinforcement learning
Keywords:
Proximal Policy Optimisation, finite-precision arithmetic, numerical stability, tanh-squashed policies, continuous-control reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WucnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the loss of action information caused by limited floating-point precision when PPO employs a tanh-squashed Gaussian policy. The authors show that once actions saturate at the boundaries, the original sampled pre-tanh latents cannot be accurately recovered, leading to biases in likelihood calculations and policy gradients during policy updates. The paper theoretically analyzes this issue and compares two solutions: retaining the original latent variables (DUAL) and computing probabilities over quantization intervals (CELL). Experiments on three MuJoCo simulation environments—HalfCheetah, Walker2d, and Ant—as well as with an independent PPO implementation demonstrate that these methods substantially reduce extreme approximate KL values and numerical instability, but do not consistently improve task returns.

Critical strengths
  1. In-depth problem analysis with clear theoretical derivations. The paper identifies the loss of action information caused by limited floating-point precision in tanh-squashed PPO and mathematically demonstrates that policy gradient bias can arise even when the initial likelihood ratio is 1. This analysis provides valuable insights into potential numerical instabilities in PPO training and offers theoretical contributions.
  2. Controlled experimental validation of numerical stability. The authors conduct comparative experiments across three MuJoCo environments and further verify the existence of the problem using an independent PPO implementation. The results show that retaining the original latent variables or using CELL likelihood can significantly reduce extreme KL divergence values and numerical anomalies, supporting the paper's proposed mechanism analysis.
Critical weaknesses
  1. Limited generalizability of the research problem and methodological novelty. This paper primarily investigates a non-default tanh-squashed gSDE PPO configuration in SB3, using a relatively large initial exploration scale (log std = 0), which causes many actions to saturate at the boundaries. The appendix experiments show that simply adjusting the exploration initialization to -2 can significantly improve the training stability and task returns of RECON. Furthermore, the validation using an independent implementation relies on artificially introducing mean shifts to induce action saturation. Therefore, the paper does not sufficiently demonstrate that this issue has a widespread impact on standard PPO configurations or real-world robotic reinforcement learning. Additionally, both DUAL and CELL are based on existing techniques, making the methodological novelty relatively limited.

  2. Insufficient validation of improvements in practical control performance. Although the paper demonstrates improved numerical stability across three MuJoCo simulation environments, the main benefits are reflected in numerical metrics such as peak approximate KL divergence, without sufficient evidence that these improvements consistently enhance actual policy performance. For example, in the Ant environment, both proposed methods achieve lower task returns than the original RECON baseline, and the best-performing method varies across tasks. Moreover, the experiments are limited to three locomotion environments with only five random seeds per setting, lacking evaluations on more complex robotic manipulation tasks or real-world robots. These limitations weaken the persuasiveness of the paper's conclusions regarding practical robotic reinforcement learning applications.

Questions for authors
  1. Regarding the generalizability of the problem: This paper primarily investigates tanh-squashed gSDE PPO with a relatively large initial exploration scale (log std = 0). Given that standard PPO does not use this configuration by default, could the authors further clarify how frequently and severely this numerical issue occurs in other commonly used squashed-policy configurations or practical robotic reinforcement learning settings?

  2. Regarding the impact of hyperparameter settings: Table 3 shows that simply adjusting the initial exploration scale from 0 to -2 can significantly improve the training stability and task returns of RECON. Could the authors further compare RECON-INIT2, DUAL, and CELL under consistent experimental settings to demonstrate that the proposed methods offer clear advantages over reasonable hyperparameter tuning?

  3. Regarding practical task performance: Although DUAL and CELL significantly reduce peak approximate KL divergence, task returns do not show consistent improvements. In particular, both methods achieve lower returns than RECON in the Ant environment. Could the authors explain this phenomenon and further clarify whether improvements in numerical stability can translate into reliable gains in policy performance?

  4. Regarding the appropriateness of evaluation metrics: The paper primarily uses peak approximate KL divergence to measure training stability. However, the authors also acknowledge that extremely large approximate KL values do not necessarily accurately reflect the true KL divergence between policy distributions. Could the authors provide additional training stability metrics, such as actual changes in policy distributions or more detailed statistics on gradient anomalies, to further validate the effectiveness of the proposed methods?

Reviewer_BXoFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a finite-precision issue in PPO, where different pre-tanh samples can produce the same stored action, leading to incorrect likelihood reconstruction and potentially incorrect policy gradients. The authors analyze this issue in Stable-Baselines3 and investigate two alternatives: DUAL, which retains the original latent, and CELL, which computes rounding-cell probabilities. They provide mathematical derivations and experiments on three MuJoCo tasks, along with numerical diagnostics and an independent reproduction. The results show substantial reductions in extreme approximate-KL values and numerical failures, although improvements in task returns are inconsistent.

Critical strengths
  1. A useful numerical observation.

The paper identifies a specific failure mode in squashed PPO likelihood evaluation. Equation (7) explains the resulting gradient discrepancy, while Section 5.1 provides direct evidence of information loss during collection. This is a useful implementation-level insight.

  1. Evidence supporting the proposed mechanism.

The paper combines mathematical analysis, direct measurements, and controlled experiments. Table 1 shows a substantial reduction in HalfCheetah's median peak approximate KL from (1.26 x 10^6) to 0.0361 under DUAL, and Section 5.4 provides an independent reproduction.

Critical weaknesses
  1. Limited novelty and practical scope.

The general problem of information loss under action transformations and its implications for policy gradients has already been studied in prior work, such as CAPG and Marginal Policy Gradients. Retaining pre-transformation samples is also an established implementation technique. The paper's main novelty lies in identifying and measuring a specific finite-precision reconstruction issue in squashed PPO. While this is useful, the contribution appears relatively incremental. More importantly, the experiments mainly investigate highly saturated configurations. It is unclear how often this issue occurs or significantly affects optimization under more typical PPO settings. This limits the significance of the contribution. A clearer distinction from prior work and stronger evidence of practical relevance would help address this concern.

  1. The empirical evaluation does not fully establish the practical benefits of the proposed alternatives.

The comparison between RECON, DUAL, and CELL is reasonable for studying the numerical mechanism, but it is less convincing as an evaluation of practical improvements. The reported gains are mainly in approximate-KL diagnostics and numerical stability. Task returns are inconsistent across environments (Table 1). For example, DUAL improves the HalfCheetah return from 233.3 to 1043.0 but performs worse than RECON on Ant (−13.4 v.s. −8.4). The return averages also exclude failed runs, making direct comparisons harder to interpret. It is therefore important to distinguish correcting likelihood reconstruction from improving PPO training or final performance. The evaluation would also benefit from direct paired comparisons of gradient errors and clipping decisions, along with better-controlled comparisons under less saturated settings. Finally, many experimental findings are difficult to verify because Figures 2, 4, and 5 contain overlapping data points without sufficiently accessible numerical results. These reporting issues further weaken the empirical evidence.

Questions for authors
  1. The paper discusses CAPG, Marginal Policy Gradients, and existing latent-retention techniques. Could the authors clarify what the main conceptual contribution is beyond these established approaches? In particular, is the novelty primarily the identification of a specific finite-precision issue in SB3, or does the proposed analysis provide broader insights into policy optimization?

  2. The main comparison focuses on RECON, DUAL, and CELL across three MuJoCo tasks. Are these comparisons sufficient to establish the practical advantages of the proposed alternatives? Since DUAL retains sampled latents while CELL evaluates a different probability event, how should their results be compared fairly? It would also help to clarify how these approaches relate to existing implementations that already retain pre-transformation samples.

  3. The results show substantial reductions in approximate KL and numerical failures, but improvements in task returns are inconsistent (Table 1). How should we interpret this difference? Does correcting the reconstruction error meaningfully improve PPO optimization, or mainly prevent numerical failures under highly saturated conditions? Evidence from less saturated configurations, together with direct gradient or clipping comparisons, would help clarify the practical significance.

  4. Many results in Figures 2, 4, and 5 are difficult to interpret because markers overlap and exact numerical values are not readily available. Could the authors provide corresponding numerical tables, ideally including individual seed results and variability? Also, Figure 5(c) places failure markers on the endpoint-return axis, whereas Section 4 states that failed runs have no endpoint return. Could the authors clarify this presentation?

  5. The overall narrative is difficult to follow in several places. Section 2 mixes background, implementation details, and related work without clearly establishing their relationships. Several figures also contain crowded annotations, overlapping elements, and insufficiently explained labels (e.g., B0/B4/B4A/B4B in Figure 2). Could the authors improve the organization and figure presentation, particularly by making the experimental configurations and the evidence supporting each main claim easier to follow?

Reviewer_HqmmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on the diagnosis and measurement of a storage-induced likelihood error in the policies with tanh-squashed actions (tanh is the standard mechanism applied to keep the values of different action dimensions within the desired range). The main target of the analysis is the Stable-Baselines3 PPO pathway. The Authors report that across three MuJoCo tasks, storing the sampled pre-tanh value substantially reduces peak approximate KL despite equal or greater boundary incidence (examples include standard environments as HalfCheetah).

Critical strengths

The paper tackles an important practical problem in designing policies with actions taken from the bounded regions: standard action normalization procedures often wipe off important signal coming from the activation vectors producing them. The paper conducts detailed studies on various variants of the PPO algorithm and provides detailed ablations for the selected RL environments (e.g. studies with wider arithmetic).

Critical weaknesses

The paper provides a very incremental novelty. It is not clear whether it produces a solution to the problem studied there. The main focus is on the diagnosis and accurate quantitative evaluation of the problem, as the Authors directly admit in the introduction. It looks like an engineering effort is limited to a very narrow set of three very standard and easy Mujoco environments (HalfCheetah, Walker2D and Ant). The very limited scope of the RL environments considered (the Reviewer would expect the Authors to try at the very least the Humanoid from the standard Mujoco suite and at least some of the environments from the DeepMind Control Suite suite) and very small number of seeds applied (five seeds for the environments that can be solved within a couple of minutes with very limited computational resources) is concerning. The introduction does not provide a clear description of what the paper achieves, what the main findings are. The impact of the AI agents (that the authors refer to in the AI Use Statement) on writing is clearly seen in the way the paper is organized.

Questions for authors

The Reviewer would like to ask the Authors to address the points raised above in the critical weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.