acceptodds
Under review as a conference paper at ICLR 2027

ISPO: Correcting Policy-Induced Value Lag in PPO via Importance-Sampled Policy Shifts

Abstract

On-policy actor–critic RL, represented by PPO, trains the critic on rollouts from the policy it just replaced. After each actor update, the true value function moves while the critic still fits the previous one, leaving a **policy-induced value lag**. Our analysis shows that this lag biases every advantage estimate the actor consumes and, when large, violates the small-update assumption that PPO relies on. The lag is bounded by the product of effective horizon, action-wise residual dispersion, and policy-update size, identifying long-horizon, action-sensitive control as its worst case. Prior work addresses this through trajectory-level reweighting (Retrace, V-trace), which compounds importance-ratio variance, or policy-conditioned values (PeVFA), which add parameters and training costs. We argue neither is necessary: PPO constrains policy updates, making the value shift nearly first-order and estimable directly from the existing rollout. Motivated by this, we propose **I**mportance-**S**ampled **P**olicy-Shift **O**ptimization (**ISPO**). We construct a **PINN-style evaluation residual** per transition based on the policy evaluation PDE, and reweight it with a **single-step importance ratio**, avoiding cumulative off-policy weights on the whole trajectory. The resulting shift corrects the critic target after each actor update toward the updated policy's value function. We evaluate ISPO on an analytic LQR system, standard MuJoCo benchmarks, and an embodied manipulation task. Experiments show that ISPO achieves higher final return while adding no environment interaction, no learnable parameters, and no cumulative importance weights, improving return by 10% to 40% over vanilla PPO and matching or exceeding other baselines. [Here is Project Page](https://anonymous.4open.science/r/ISPO-FFF6/)

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, PPO, PDE, Physics-Informed Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NS5wOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

On-policy algorithms update the policy using the value function of the previous policy, which may cause the value-lag issue. However, learning a separate new value function increases the computation burden. This paper proposes an approach that leverages the first-order information of the value network to correct value estimates without additional rollout or learnable parameters.

Critical strengths

This paper studies an interesting and important problem in on-policy RL. Its presentation is clear and easy-to-follow. I did not check every detail of its theory proofs, but their claims seem reasonable to me. Empirical results are comprehensive and look good.

Critical weaknesses

Figure 5 compares the performance of different methods on MuJoCo tasks, where the training curve of many several methods are still increasing while being terminated after 1M steps. It would be more convincing to run more env steps and compare algorithms when their returns get stable.

One minor thing: I would recommend rewriting the policy-parameter θˉ\bar{\theta} as Δθ\Delta \theta or something else, since θˉ\bar{\theta} is often referred to some fixed or initial parameters.

Questions for authors
  • In Equation 17, it differentiates the value network with respect to its input to obtain first-order information. Are there any risks that this operation may cause numerical instability?

  • Table 6 shows that the correction scale α\alpha varies in different environments. Can we find a way to automatically tune it?

3 comments

We sincerely thank the reviewer for recognizing the importance of the problem, the clarity of our presentation, and the soundness of our theoretical analysis. We appreciate your constructive suggestions and address them below.


[W1] Longer Training Runs

Thank you for pointing this out. We reran the experiments on Walker2D and Ant with the number of steps increased to 3×1063 \times 10^6. You can find our latest results in the revised manuscript.

[W2] Notation for the Policy Update

Thank you for your suggestion. We have switched to Δθ\Delta\theta.

[Q1] Numerical Stability of Value Gradients

We believe this question has two aspects. The first is whether the gradients themselves can explode numerically. The second is whether the gradients can accurately capture first-order changes in the value shift.

First, foundational studies from PINNs [1] have shown that gradient stiffness associated with PDE residuals typically depends on the properties of the equation itself. In particular, higher-order derivatives or large coefficients can substantially increase gradient magnitudes. The HJB equation, however, involves only first-order spatial derivatives. The coefficient of the derivative term is determined by the system dynamics, and the term itself remains modest in magnitude in continuous action spaces. The equation therefore avoids the additional gradient amplification associated with higher-order differentiation.

Second, whether the gradients can accurately capture first-order changes in the value shift ultimately depends on the accuracy of the critic’s estimates. Early in training, the critic has yet to converge sufficiently, so the estimated value shift may be considerably noisy. As training stabilizes, the critic’s estimates become more accurate, and the value-shift correction begins to take effect. This trend is consistent with the training behavior observed in Figure 5, particularly on Swimmer, where our method begins to yield sustained improvements in the middle to later stages of training.

[Q2] Automatic Tuning of the Correction Scale

We agree that automatically adapting α\alpha would be a valuable extension. One promising direction is to adjust the correction magnitude according to online estimates of the value-shift signal and its uncertainty, allowing stronger corrections when the estimate is reliable and more conservative corrections when its variance is high. These hypotheses involve methodological changes and require further experimental evidence for validation, so we leave them for future discussion.


Manuscript changes:

  • We updated the Walker2D and Ant learning curves in Figure 5 and the corresponding results in Table 1, under 3×1063 \times 10^6 steps.
  • We replaced θ^\hat{\theta} with Δθ\Delta\theta.

We hope these responses address your main concerns. Please kindly consider changing the rating if our responses have addressed your concerns.


References:

[1] Wang, S., Teng, Y., and Perdikaris, P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SISC. 2021.

I would like to thank the authors for their responses and revisions, they resolved most of my concerns. Regarding the training duration, I recommend the authors to extend the horizon for all tasks in a future version, especially those in which the training curves are still climbing. Other than that, I have no further questions and still vote for acceptance.

Thank you for maintaining your positive assessment! Regarding extended training, the experiments take approximately 19 hours per benchmark on a single RTX 5090. We have been running these tests and expect to include the corresponding results in the final version.

Reviewer_DhKPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies policy-induced value lag in standard PPO, where, after an actor update, the critic still approximates the value of the behavior policy rather than the updated policy. The authors claim that this lag distorts the advantages used to update the actor in the next epoch. ISPO, proposed by the authors, adds a correction term αΔ^\alpha\hat\Delta to the critic's regression target and then recomputes GAE using the corrected critic. Δ^\hat\Delta is obtained from the existing rollout by accumulating, backward along each trajectory, the single-step ratio deviation (clip(ρ)−1)(\mathrm{clip}(\rho)-1) multiplied by the old critic's one-step residual. The method is evaluated on an LQR system, showing that the lag exists and Δ^\hat\Delta reduces value error, as well as on six MuJoCo tasks and four LIBERO tasks, where ISPO is compared against PPO and five other baselines.

Critical strengths

ISPO is a simple plug-in method that does not introduce additional parameters or interactions.

The experiments are comprehensive, covering a synthetic environment, MuJoCo tasks, and embodied tasks.

Critical weaknesses

The PINN framing does not seem essential. The residual in Eq. (17) is a first-order approximation of the TD error divided by sˉ\bar s, and Appendix F.2 shows only marginal gains from using the PDE over TD residuals.

A simpler alternative is not considered: training the critic on the current batch and then computing GAE before updating the actor. This would remove the lag using real data without approximation.

α\alpha is tuned separately for each environment (from 0.25 to 3), and the value used for Ant coincides with the best value in its ablation. α=1\alpha=1, as implied by the derivation, underperforms PPO on Ant, while α=3\alpha=3 on Swimmer cannot be explained as a lag correction.

Questions for authors

Could the authors empirically or theoretically demonstrate that ISPO outperforms the simpler alternative raised in the second point of the Weaknesses?

What are the results when α\alpha is fixed at 1 across all tasks?

5 comments

We sincerely thank the reviewer for your constructive feedback and for recognizing the simplicity of our method and the breadth of our experimental evaluation. We respond to each of your comments in detail below and hope our responses address your concerns.


[W1] Role of the PDE Framework

We would like to clarify that the primary contribution of this paper is to establish the link between policy changes and value shifts using a PDE framework, while the PINN-style residual is a method that follows naturally from this theoretical analysis.

RL is increasingly applied to control tasks in continuous state and action spaces, such as embodied reinforcement learning. Compared with conventional discrete-time TD analysis, we therefore believe that the main value of the PDE framework based on the HJB equation lies in providing a theoretical means of understanding and characterizing how policy changes affect the value function from a continuous-space perspective. Its value thus goes beyond constructing an alternative to the TD residual. For example, Proposition 4.4 shows how the sensitivity of the system dynamics affects the spatial gradient of the value shift, which in turn affects how accurately the critic evaluates different actions (as illustrated in Figure 3). This result is difficult to derive from a discrete TD perspective.

As for ISPO itself, we use the PDE residual to compute the value shift because it reduces dependence on how the value function changes nonlinearly over a finite step, thereby reducing the impact of errors in the critic's curvature estimates. It is also closely aligned with our theoretical derivation, which makes it our preferred approach. We believe that even using a TD version of ISPO would preserve the benefit of "avoiding full trajectory reweighting" stated in our second contribution, because the choice is only an implementation-level trade-off.

[W2] GAE Refresh and Policy-Induced Value Lag

The reviewer's suggestion to "training the critic on the current batch and then computing GAE before updating the actor" is actually already part of our method, as shown in Lines 11 and 13 of Algorithm 1. (Note that Lines 4 through 14 form a loop. Recomputing GAE at the end of the loop is therefore equivalent to refreshing GAE before the actor update in the next iteration.)

We believe the reviewer may be conflating critic approximation error with policy-induced value lag. The former is the approximation error of the critic network, while the latter arises from PPO's update mechanism itself. According to Eq. (5), the HJB equation, each policy in on-policy RL corresponds to a unique value function. And as shown in Figure 1, PPO first uses the policy trained in the previous iteration, πt−1\pi_{t-1}, to collect rollout data and then uses those data to update the critic. At this point, the critic learns the value function Vπt−1V^{\pi_{t-1}}. As the actor is updated, the value function VπV^\pi corresponding to the policy π\pi continues to change, while the critic continues to fit Vπt−1V^{\pi_{t-1}} using rollouts collected under the old policy πt−1\pi_{t-1}.

Therefore, the reviewer's suggestion that "computing GAE before updating the actor can eliminate the lag using actual data without approximation" does not resolve this issue. Even if the critic is trained first and GAE is then recomputed within the loop, the critic still learns the value Vπt−1V^{\pi_{t-1}} under the old policy πt−1\pi_{t-1} because the rollouts used to train it were collected under that policy. Obtaining the value of the current policy would require re-interact with the environment and collecting new rollouts within the loop, which would run counter to PPO's deliberate reuse of data from the old policy for multiple update epochs. Therefore, our method directly corrects the critic toward the value of the current policy without collecting new rollouts.

[Q1] ISPO vs. GAE Refresh Alone

The ablation study in Section 6.3 shows that refreshing GAE alone (i.e., setting α=0\alpha=0) is insufficient to achieve the best performance. The value-shift correction is needed to obtain the full improvement from ISPO. Our revised manuscript includes ablation results across three environments. The consistent results across these environments confirm that ISPO yields further improvements over the GAE refresh scheme you proposed.

[W3 & Q2] Tuning α\alpha and Results with α=1\alpha=1

Thank you for raising this point. We would like to emphasize that α=1\alpha=1 in the theoretical derivation corresponds to the exact value shift under ideal conditions, whereas Eqs. (17)-(19) yield only an approximate estimate of that shift in practice. This estimate is inevitably affected by factors such as the finite-step approximation and critic error. The purpose of α\alpha is therefore to calibrate the magnitude of the approximate estimate and provide a manually tunable parameter for balancing bias and variance. This follows the same principle as adjusting hyperparameters such as γ\gamma and the clipping threshold across tasks in other RL algorithms and is a practical engineering compromise.

We did not conduct a fine-grained sweep over α\alpha to select the optimal value for all environments. In the main experiments, we used the default value of 11 for Hopper, Walker2D, and Reacher. For Swimmer, the forward-difference estimate of the dynamics is small. A small α\alpha would make the correction in Eq. (20) nearly vanish, so we increased α\alpha to compensate.

The results with α\alpha fixed at 11 vary by environment. For example, in our ablation study in Section 6.3, setting α=1\alpha=1 still yields substantial improvements on HalfCheetah and Hopper. On Ant, its performance is like that of standard PPO. In all three environments, however, the best performance is achieved with α<1\alpha<1. This indicates that the estimation error introduced by Eq. (20) varies across environments, so the results should be interpreted for each environment. At this stage, we have made a reasonable and practical trade-off to address the gap between theory and implementation, and we leave the automatic tuning of α\alpha to future work.


Manuscript changes:

  • We added ablation experiments on HalfCheetah and Hopper to Section 6.3.

We hope these responses address your main concerns. Please kindly consider changing the rating if our responses have addressed your concerns.

Thank you for the detailed response and the additional ablations on HalfCheetah and Hopper, which are helpful. Some of my concerns remain.

W1. I appreciate the clarification that the TD and PDE residuals are an implementation-level choice; this confirms that the method does not depend on the PDE framing. Regarding Proposition 4.4, it requires a continuous state space and differentiable dynamics, but not continuous time. For a discrete-time MDP with a closed-loop transition map of Lipschitz constant LL, the same argument gives ∥∇Δ∥≤∥∇g∥/(1−γL)\|\nabla\Delta\| \le \|\nabla g\|/(1-\gamma L) for γL<1\gamma L < 1. Proposition 4.4 is also not used by the algorithm or tested empirically.

W2. My suggestion is not part of Algorithm 1. Lines 11 and 13 fit the critic after the actor update and refresh GAE for the next epoch, so the first actor epoch of every iteration still uses a critic that has never seen the current batch. I suggested fitting the critic on the current batch, ideally on held-out trajectories, before the first actor update. I am not conflating approximation error with lag: I agree this yields Vπt−1V^{\pi_{t-1}}, and my point is that Vπt−1V^{\pi_{t-1}} is exactly what PPO’s surrogate requires (as also noted by Reviewer M19h). The part of the lag that departs from PPO’s theory, namely that the critic lags behind the rollout policy when advantages are computed, is removed by this alternative using real data. Targeting VπtV^{\pi_t} within an iteration still needs justification. The grasping example in the response to M19h does not provide it: with old-policy advantages, the first-order surrogate estimates the improvement as 3.2+0.4×8=6.43.2 + 0.4 \times 8 = 6.4, versus the true 7.047.04. The remaining 0.640.64 is the second-order state-distribution term, which ISPO’s Δ^\hat\Delta, also first order along the old trajectories, neglects as well. The example therefore shows policy improvement rather than an error in the advantages, and it does not distinguish ISPO from PPO’s own approximation. Comparing against α=0\alpha = 0 does not answer Q1, since GAE refresh after the actor update is not the alternative I described.

W3. The explanation for Swimmer suggests that α\alpha compensates for a mis-scaled estimator rather than calibrating an accurate one: if Δ^\hat\Delta is small because the lag is small, scaling it by 3 goes beyond the derived correction. In addition, the 13/15 paired comparison uses the best α\alpha from the ablation (e.g., 0.75 on HalfCheetah and Hopper, while the main results use 0.5 and 1), selected on the same five seeds. Results with a pre-specified α\alpha, or on fresh seeds, would be more convincing.

I therefore maintain my score.

Reviewer_M19hOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper studies how a policy update changes the value function that the critic should track. ISPO estimates this change from the existing rollout, using a local value residual and one importance ratio per transition. It corrects the critic targets and refreshes the advantages. The analysis relates the value shift to horizon, action sensitivity, and policy change. An analytical LQR example checks the mechanism; MuJoCo and LIBERO experiments test the algorithm.

Critical strengths
  1. The correction has a clear intuition: estimate which actions the new policy favors, then accumulate their local effect on value. Equations (17)–(20) turn this into a simple addition to PPO, without extra environment interaction or a learned dynamics model.
  2. The research questions at lines 349–353 clearly organize the experiments around the mechanism and its limits. The experiments include meaningful controls. The LQR example separates policy-induced value changes from critic fitting errors (lines 357–377). Higher mean returns than PPO on five of six MuJoCo tasks support the practical contribution. The refresh-only and TD-residual comparisons, together with the disclosed unsuccessful cases, help assess what the method adds.
Critical weaknesses
  • The assumptions and meaning of the correction need to be clearer in the main text. What does “sufficient regularity” at line 220 require? Appendix B supplies conditions, but their practical meaning should accompany Lemma 4.2 and Corollary 5.1, including which are expected to hold in the experiments. The idealized derivations appear sound under those conditions. I am less sure how the value-shift analysis establishes the advantage bias or violation of PPO's small-update assumption claimed at lines 14–15 (abstract) + Appendix C. PPO deliberately uses old-policy advantages (and this is safe and sound, for a theoretical analysis, see [1]). Equation (49) shows that changing the critic changes TD feedback, but which target makes this difference an estimation error? There is also a gap between the idealized and implemented corrections: Appendix D excludes clipping, whereas Eq. (18) clips the ratio. I think the cancellation at line 305 then need not hold. These points limit the claimed explanation of why the actor update improves.

  • The evidence for the correction's separate benefit remains uncertain. Table 3 reports Ant returns of 1473 for PPO, 1694 with GAE refresh, and 1987 with the selected correction. The variation alone cannot settle whether the added gain is reliable. More consequentially, the selection procedure for the task-specific scales in Table 6 is unclear (and environment-dependent, whereas all the parameters remain the same for PPO). How much of the gain survives a comparison with matched tuning? This matters for the attribution at lines 472–475.

[1] Mirror Learning: A Unifying Framework of Policy Optimisation (ICML 2022). Jakub Grudzien Kuba, Christian Schroeder de Witt, Jakob Foerster

Questions for authors
  1. Which advantage or policy-gradient target does the refreshed estimator aim to estimate, and how does Appendix C show reduced error relative to that target?

  2. Which assumptions in Corollary 5.1 and Appendix D are expected to hold experimentally? How often is ratio clipping active, and how does clipping affect the claimed zero-mean cancellation of critic bias?

  3. How were correction scales selected, including seeds and baseline tuning budgets? Could paired differences from the existing runs clarify Tables 3 and 9?

  4. For LIBERO, what is the critic input in Eq. (17), how are its state gradient and transition difference computed, and how many seeds and evaluation episodes support Table 2?

6 comments

We sincerely thank the reviewer for your thoughtful and constructive feedback and for recognizing the breadth of our experiments. We address your main concerns below.


[W1.1] Regarding assumptions for Lemma 4.2.

We thank the reviewer for the suggestion. We have added a brief description of the assumptions to the main text (Lines 219-220). The phrase "sufficient regularity" in Line 220 refers primarily to Assumptions B.4–B.6 in Appendix B.2:

  • Assumption B.4 requires the value shift to be sufficiently smooth, which is a prerequisite for our analysis in continuous spaces. We use tanh⁡\tanh as the activation function (Table 5), which ensures that the critic's estimates are smooth.
  • Assumption B.5 requires the contribution of the discounted terminal value shift to vanish as the time horizon tends to infinity. We use a discount factor γ<1\gamma<1, as is standard in most RL settings.
  • Assumption B.6 requires the discounted local value mismatches accumulated along a trajectory to be integrable. This ensures that their cumulative contribution is finite and that the integral in Eq. (13) is well defined.

[W1.2] Regarding assumptions for Corollary 5.1.

Corollary 5.1 inherits the assumptions of Lemmas 4.1 and 4.2 and additionally requires smoothness with respect to the policy parameters. Appendix D analyzes the correction under small policy updates. The main conditions relate to our experimental setup as follows.

  • Assumption B.1 requires the updated policy to be continuous with respect to the behavior policy. The stochastic policies in MuJoCo and the noise injection in LIBERO support a common action support across consecutive updates.
  • Assumption B.2 requires finite second moments for the importance ratios and evaluation residuals. We expect the ratio moments to be well behaved when consecutive policies are close. On the sampled transitions, the residual is computed from the observed reward, the state increment, and the output and input gradients of the frozen critic.
  • Assumptions B.4–B.6 carry over from Lemma 4.2, as discussed in [W1.1]. In MuJoCo, we use a discount factor of γ=0.99\gamma=0.99, so the weights on the terminal value shift and on later local mismatches decay geometrically with the horizon.
  • Corollary 5.1 additionally requires the policy and its value function to be twice continuously differentiable with respect to the policy parameters and allows differentiation under the discounted expectation. The tanh⁡\tanh actor and critic in MuJoCo provide smooth policy and critic parameterizations.
  • Appendix D assumes that the importance ratios stay close to one for the first-order trajectory approximation. In MuJoCo, we use a PPO clipping threshold of ϵ=0.2\epsilon=0.2 and an actor learning rate of 3×10−43\times10^{-4}. In LIBERO, the actor learning rates range from 5×10−65\times10^{-6} to 2×10−52\times10^{-5}. These settings are designed to keep consecutive policy updates small.

[W1.3] How the value-shift analysis establishes the advantage bias or violation of PPO's small-update assumption?

We agree with the reviewer that "PPO intentionally leverages the advantages of the old policy", and the key to achieving this is importance sampling (IS). Consider an action aa that has different probabilities under the new policy πt\pi_t and the old policy πt−1\pi_{t-1}. IS assigns different weights to the advantage AA according to the probability ratio between the two policies, while AA is computed using the critic. Please note that, at this stage, the critic essentially approximates the value Vπt−1V^{\pi_{t-1}} from the rollout under old policy πt−1\pi_{t-1}, rather than VπtV^{\pi_t} under the current policy. Therefore, this actually involves a hidden assumption that Vπt−1V^{\pi_{t-1}} and VπtV^{\pi_t} should not differ significantly. Otherwise, IS merely assigns a larger weight to an advantage AA computed from an inaccurate value estimate, which can introduce significant noise. PPO intentionally constrains the magnitude of policy updates to be small, which can generally simultaneously limit the value shift. However, our paper precisely aims to investigate under what circumstances Vt−1V^{t-1} and VtV^t can differ substantially, even when the magnitude of policy updates is small, which we called action-sensitive value.

To illustrate this issue, we consider a numerical example:

Consider a robotic grasping scenario (as illustrated in Fig. 3). Suppose that a successful grasp yields a reward of 100, while failure yields 0. At step ss, the robot must first execute an action asa_s to approach the object and then complete the grasp in the following step. Under the old policy πt−1\pi_{t-1}, the probability of selecting the correct action at each of the two steps is 40%, while the new policy πt\pi_t increases this probability to 48%. Thus, the importance ratio at each step is only 1.21.2, which remains within the clipping range.

However, assuming that a successful grasp requires selecting the correct actions in both consecutive steps, and ignoring discounting, the value under the old policy is Vπt−1=0.42×100=16V^{\pi_{t-1}}=0.4^2\times 100=16, while the value under the new policy is Vπt=0.482×100=23.04V^{\pi_t}=0.48^2\times 100=23.04. Although the action probability at each step increases by only 8%, the value shift reaches 7.04, corresponding to 44% of the old policy's value. In this case, IS effectively assigns a higher weight to the Advantage computed under the old policy, but cannot correct the discrepancy of up to 44%. Instead, it may further amplify discrepancy in policy updates.

This example above illustrates that PPO’s IS cannot correct systematic errors arising from value shifts in tasks with outcomes that are sensitive to small changes in actions. Such scenarios are common in embodied AI. For example, in humanoid locomotion tasks, even a slight change in an action can cause the robot to fall, substantially reducing the reward it receives.

[W1.4 & Q2] Regarding clipping.

Regarding clipping in Eq. (18), we would like to clarify that the theoretical analysis and the practical algorithm should be considered separately. In the theoretical derivation, we use the unclipped importance ratio, so the zero-mean property holds exactly. The first-order approximation analysis in Appendix D is also independent of clipping, so introducing clipping in the practical algorithm does not affect the theoretical conclusions we have established. During training, clipping serves primarily as a safeguard for numerical stability. It prevents extreme values from producing excessively large corrections that could cause training to collapse. Like importance sampling in PPO, this is also a practical strategy for balancing bias and variance.

We added an experiment to measure how often clipping is triggered in all environments, as shown in the table below.

Table W1.1: ISPO Clipping Frequency

EnvironmentISPO clipping frequency (%)
HalfCheetah11.74%
Swimmer3.15%
Hopper8.28%
Walker2D9.14%
Ant6.23%
Reacher7.39%

Across the six environments, the clipping was triggered for 3.15% to 11.74% of the evaluated ratios. Thus, ISPO used the original importance ratio in at least 88.26% of cases in every environment. Clipping serves as a safeguard for the minority of ratios outside the prescribed interval, limiting the magnitude of their contribution to the value correction.

[W2] About correction's separate benefit.

Thank you for raising this point. Regarding the hyperparameters in Table 6, α\alpha and ϵv\epsilon_v are specific to ISPO and are absent from PPO. PPO and ISPO use identical settings for all shared hyperparameters. The comparison is therefore fair.

Regarding α\alpha. We did not conduct a fine-grained sweep over α\alpha to select the optimal value for all environments. In the main experiments, we used the default value of 11 for Hopper, Walker2D, and Reacher. For Swimmer, the forward-difference estimate of the dynamics is small. A small α\alpha would make the correction in Eq. (20) nearly vanish, so we increased α\alpha to compensate.

We would like to emphasize that α=1\alpha=1 in the theoretical derivation corresponds to the exact value shift under ideal conditions, whereas Eqs. (17)-(19) yield only an approximate estimate of that shift in practice. This estimate is inevitably affected by factors such as the finite-step approximation and critic error. The purpose of α\alpha is therefore to calibrate the magnitude of the approximate estimate and provide a manually tunable parameter for balancing bias and variance. This follows the same principle as adjusting hyperparameters such as γ\gamma and the clipping threshold across tasks in other RL algorithms and is a practical engineering compromise. We added two sets of ablation experiments, one on HalfCheetah and one on Hopper, to show how different values of α\alpha affect performance across environments. The results are reported in the updated Section 6.3. We believe the additional experiments help verify that the performance gains extend to more environments.

The results with α\alpha fixed at 11 (matched tuning) vary by environment. For example, in our ablation study in Section 6.3, setting α=1\alpha=1 still yields substantial improvements on HalfCheetah and Hopper. On Ant, its performance is similar to that of standard PPO. In all three environments, however, the best performance is achieved with α<1\alpha<1. This indicates that the estimation error introduced by Eq. (20) varies across environments, so the results should be interpreted for each environment. However, the results across environments show that including a correction term in a reasonable range consistently outperforms refreshing GAE alone. We therefore argue that this a reasonable engineering trade-off. The performance gains still come from the correction term in Eq. (20), so they cannot be attributed solely to adding extra parameters to PPO.

[Q1] Which advantage or policy-gradient target does the refreshed estimator aim to estimate?

Thank you for raising this question. According to Eq. (5) (the HJB equation), each policy in on-policy RL has its own value function. As shown in Figure 1, PPO first collects rollout data using the policy πt−1\pi_{t-1} trained in the previous iteration and uses these data to update the critic. At this point, the critic learns the value function Vπt−1V^{\pi_{t-1}}. As the actor is updated, the value function VπV^{\pi} associated with the policy π\pi also changes, while the critic continues to fit the value function Vπt−1V^{\pi_{t-1}} of the old policy πt−1\pi_{t-1}. This means that even if the critic learns Vπt−1V^{\pi_{t-1}} exactly, its estimate can still differ from VπtV^{\pi_t}, the value function of the updated policy πt\pi_t. Our updated estimator is designed to directly approximate the current policy's value function VπtV^{\pi_t} without collecting new rollouts.

As explained in our response to [W1.3], PPO's importance sampling leaves the value lag uncorrected, and this lag introduces error into the actor update. By directly correcting the value estimate from Vπt−1V^{\pi_{t-1}} toward VπtV^{\pi_t}, our method reduces this estimation error at its source. Figure 6 demonstrates this effect: ISPO's value estimates more closely match the Monte Carlo estimates used as ground truth, effectively correcting PPO's underestimation of the current policy's value.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful reading and constructive feedback. M19h highlighted the intuition behind the correction, the experimental design, and the controls that help identify where ISPO is effective. DhKP appreciated the simplicity of adding ISPO to PPO and the breadth of the evaluation across LQR, MuJoCo, and embodied manipulation. NS5w recognized the importance of policy-induced value lag in on-policy reinforcement learning and found the paper clear and the theoretical claims reasonable.


Below, we summarize the main concerns of the reviewers and our responses:

What value lag means for PPO’s advantage estimates (M19h, DhKP)?

We clarified that value lag can arise even when the critic perfectly estimates the rollout policy’s value. Once the actor updates, refreshing GAE with a critic still fitted to the rollout policy leaves the same value mismatch. Therefore, ISPO uses the existing rollout to move the critic toward the updated policy’s value before refreshing GAE.

We clarified the assumptions in Sections 4.2 and 5 and expanded Appendix C to show how the mismatch affects GAE on the same rollout. The theoretical cancellation uses unclipped importance ratios.

What the PDE framework contributes (DhKP).

The PDE analysis under HJB equation characterizes how policy updates change the value function and how sensitivity in the system dynamics shapes the spatial gradient of that change (Proposition 4.4). This conclusion can only be derived within a framework with continuous state and action spaces, which is why we use a PDE framework. ISPO’s local residual follows directly from this analysis, and the choice between continuous and discrete residuals is only an implementation detail. We therefore believe that our second contribution holds whether we use the TD residual or the PDE residual.

Whether the correction adds value beyond GAE refresh and how much tuning matters (M19h, DhKP, NS5w).

We added HalfCheetah and Hopper ablations in Section 6.3, using the same five seeds and identical settings for all shared hyperparameters. At the selected correction scales, ISPO improved on GAE refresh alone in 13 of 15 matched runs across HalfCheetah, Hopper, and Ant. Therefore, the correction improves on GAE refresh across multiple environments, with gains that depend on its scale.

The scale α\alpha aims to control how much of the estimated value shift enters the critic target. The theoretical correction corresponds to α=1\alpha=1 under ideal conditions, but in practice, finite-step approximations and critic error can affect the magnitude of the estimate. Calibrating the correction with α\alpha is therefore a practical compromise between the idealized derivation and its implementation, consistent with the common practice in other RL of tuning hyperparameters to balance bias and variance.

Training duration and reproducibility (NS5w, M19h).

We extended Walker2D and Ant training from one to three million environment steps. At the longer budget, ISPO achieved mean returns 9.7% and 15.4% above standard PPO, respectively (Table 1 and Figure 5). We also documented the LIBERO critic inputs, the computation of the proprioceptive gradient and transition difference, and the evaluation protocol in Appendix E.3. The reported ISPO results use one training seed and 1000 evaluation episodes per suite.


Summary of manuscript revisions

  • Theory (Sections 4.2 and 5, Appendix C). We clarified the assumptions behind Lemma 4.2 and Corollary 5.1, adopted Δθ\Delta\theta for the policy update, and expanded the analysis of how policy-induced value lag affects GAE.
  • Main experiments (Section 6.2, Figures 5–6, Table 1). We updated the Walker2D and Ant results with runs of 3×1063 \times 10^6 environment steps and clarified the comparison between critic predictions and Monte Carlo value estimates on HalfCheetah.
  • Ablations (Section 6.3, Table 3). We added HalfCheetah and Hopper to the correction-scale ablation alongside Ant, including the GAE-refresh-only baseline (α=0\alpha = 0) and the fixed α=1\alpha = 1 setting.
  • Reproducibility (Appendices E.2–E.3). We clarified the MuJoCo training budgets, identified the ISPO-specific hyperparameters, and documented the LIBERO critic inputs and how we compute the value gradient and finite-difference estimate.
AuthorsOpenReview ↗

Summary of the Discussion

Dear all reviewers and AC:


We sincerely thank all three reviewers for their careful evaluation and constructive feedback. We also thank AC for its evaluation of this paper. As the rebuttal period draws to a close, we summarize each reviewer’s assessment and indicate which concerns they considered resolved.

Strengths recognized by the reviewers

  • M19h highlighted the intuitive value-shift correction and the experimental controls that isolate its effect. In particular, the analytical LQR example separates policy-induced value lag from critic fitting error, while the GAE-refresh and TD-residual ablations help identify where ISPO adds value. After the rebuttal, M19h raised their rating to clear acceptance.
  • DhKP recognized that ISPO is a simple addition to PPO that requires no extra environment interactions or learnable parameters, and highlighted the breadth of the evaluation across LQR, MuJoCo, and LIBERO. DhKP also found the additional HalfCheetah and Hopper ablations helpful.
  • NS5w recognized policy-induced value lag as an important problem in on-policy RL, found the paper clear and the theoretical claims reasonable, and praised the breadth of the empirical evaluation. NS5w maintained their recommendation to accept.

Core concerns and responses

ConcernReviewerStatus and response
Advantage target and PPO convergence guaranteesM19h, DhKPResolved. We clarified that ISPO uses the existing rollout to estimate the updated policy’s value before refreshing GAE. We revised the abstract to describe the effect on subsequent advantage estimates more precisely and added the grasping example in Section 4.3 to illustrate the value shift.
Theoretical assumptions and importance-ratio clippingM19hResolved. We clarified the assumptions in Sections 4.2 and 5 and explained that exact cancellation in expectation uses unclipped ratios.
Benefit beyond GAE refresh aloneM19hResolved. We added HalfCheetah and Hopper ablations in Section 6.3 and reported paired comparisons using the same five seeds and shared hyperparameters. The results demonstrated the continuous improvement of ISPO over merely refreshing GAE.
Correction-scale sensitivityDhKPAddressed. The use of α\alpha is a reasonable engineering trade-off that is consistent with our theoretical derivation. Using the best tested correction scale α\alpha for each task, ISPO outperformed GAE refresh alone in 13 of 15 paired runs across HalfCheetah, Hopper, and Ant.
Role of the PDE frameworkDhKPAddressed. For the continuous control problem studied here, the continuous-time PDE framework provides a more natural setting for the derivation and serves solely as an analytical tool. The method and conclusions are independent of this choice of framework.
Critic-first baselineDhKPAddressed. The reviewer proposed critic-first schedule leaves the value lag that arises across successive PPO actor epochs unresolved, and ISPO outperformed this baseline on both tasks in our additional experiments.
LIBERO reproducibilityM19hResolved. Appendix E.3 documents the critic inputs, the gradient and finite difference in proprioceptive feature space, and the evaluation hyperparameters.
MuJoCo training durationNS5wResolved. We extended Walker2D and Ant from 1M to 3M environment steps. At 3M steps, ISPO exceeded PPO’s mean return by 9.7% and 15.4%, respectively (Table 1 and Figure 5).
Policy-update notationNS5wResolved. We adopted Δθ for the policy update.

At this point in the discussion, we have responded to all concerns raised by the reviewers. We are awaiting Reviewer DhKP’s reply to our second round of clarifications, and all other concerns have been fully resolved.


Sincerely,

Authors of Submission 14628

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.