acceptodds
Under review as a conference paper at ICLR 2027

FERPO: Forward Entropy-Regularized Policy Optimization

Abstract

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback–Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).

Primary area:
reinforcement learning
Keywords:
On-policy reinforcement learning, maximum-entropy reinforcement learning, policy optimization, continuous control.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_96Y9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FERPO is an on-policy maximum-entropy RL method that constructs a regularized target action distribution and fits the actor through forward KL. It uses self-normalized importance sampling and adaptive ESS control, avoiding critic action gradients. Experiments on MuJoCo Playground and ManiSkill show competitive performance, sample-efficiency gains on some tasks, and reduced update costs.

Critical strengths

The method provides a practical actor update using critic values without requiring action derivatives. The entropy and ESS controls are well explained, and the limitations of the theoretical guarantees are clearly stated. The evaluation covers 34 tasks, with useful ablations and runtime measurements. Competitive results and lower update costs suggest practical value for parallel on-policy learning.

Critical weaknesses

The contribution needs clearer positioning against closely related work. MORE already derives the same entropy- and KL-constrained target form, while MPO uses forward-KL policy fitting. MPO is discussed, but MORE and the on-policy V-MPO method are missing. The soft critic and ESS controller provide differences, but their added value over existing policy-fitting methods is not sufficiently established by direct comparisons. Evidence for the main motivation remains limited. The toy example uses an exact critic, so it does not test robustness to critic-gradient errors. With rollout-policy sampling, it recovers the preferred direction in only 26.4% of batches even at K = 1,024. Although this limitation is acknowledged, it remains unclear whether the practical small-K algorithm achieves the proposed exploration benefit.

Questions for authors
  1. Could you clarify the contribution relative to MORE and MPO/V-MPO, and provide a matched comparison that isolates the benefit of FERPO’s additional design choices?

  2. With the practical candidate budget and adaptive ESS control, how closely do sampled updates match a high-sample reference? What evidence connects the benchmark gains to improved exploration or reduced sensitivity to critic-gradient errors?

4 comments
AuthorsOpenReview ↗

Answer to Review 1/3

We thank the reviewer for their constructive feedback on the contribution, comparisons, and sampled updates.

W1 / Q1: Contribution and comparisons with MORE, MPO, and V-MPO

Clarify FERPO’s contribution relative to MORE and MPO/V-MPO, and provide comparisons supporting its design choices.

A1: We now discuss MORE [1] and V-MPO [2] in Section 4 (lines 260–277) and Appendix A.6 (lines 2090–2110), explaining the shared target and differences in policy fitting, critic learning, and regularization.

Relation to MORE and REPPO. MORE derives the optimal entropy- and KL-regularized target distribution. Writing its objective in policy notation, this target is

\pi_0(a\mid s)^{\lambda/(\lambda+\tau)} \exp\\!\left(\frac{Q(s,a)}{\lambda+\tau}\right),$$ where $\pi_0$ is the old policy, $\lambda$ controls KL regularization, and $\tau$ the entropy term. For fixed critic and multipliers, the corresponding policy optimization is equivalent to reverse-KL fitting: $$\begin{aligned} \min_\theta\\; (\lambda+\tau)\\, \mathbb E_{s\sim d}D_{\mathrm{KL}}\\!\left(\pi_\theta(\cdot\mid s)\\|p^*(\cdot\mid s)\right) \quad\Longleftrightarrow\quad \max_\theta\\;\mathbb E_{s\sim d}\Big[ &\mathbb E_{a\sim\pi_\theta}[Q(s,a)]\\\\ &+\tau\mathcal H(\pi_\theta(\cdot\mid s)) -\lambda D_{\mathrm{KL}}(\pi_\theta\\|\pi_0) \Big]. \end{aligned}$$ A modern neural-network adaptation of MORE would optimize this reverse-KL objective, whereas FERPO fits the target using forward KL, $D_{\mathrm{KL}}(p^*\\|\pi_\theta)$. See [1], Section 2. REPPO is related to this regularized expected-value optimization, but uses the opposite trust-region direction, $D_{\mathrm{KL}}(\pi_{\mathrm{old}}\\|\pi_\theta)$, and switches between the maximum-entropy objective inside the KL bound and a KL-only penalty outside it (Appendix A.6, lines 2103–2105). This switch brings REPPO’s loss shape closer to FERPO’s in Figure 6. MORE and FERPO still use different KL directions and can produce different updates despite their shared target. **Relation to V-MPO.** V-MPO normalizes weights across the full batch of state–action pairs, whereas FERPO normalizes candidate-action weights separately within each state. This changes how strongly each state contributes to policy fitting. To isolate this difference, hold the conditional target $p^*(a\mid s)$ fixed. Let $d(s)$ be the rollout state distribution, $Z(s)$ the target's state-dependent normalizer, and $$\ell_s(\theta) =-\mathbb E_{a\sim p^*(\cdot\mid s)} [\log\pi_\theta(a\mid s)]$$ the fitting loss at state $s$. Writing $L_{\mathrm{V\text{-}MPO}}$ for the batch-normalized fitting objective in this comparison gives $$\boxed{ \begin{aligned} L_{\mathrm{FERPO}}(\theta) &=\mathbb E_{s\sim d}[\ell_s(\theta)],\\\\ L_{\mathrm{V\text{-}MPO}}(\theta) &=\mathbb E_{s\sim d}\\!\left[ \frac{Z(s)}{\mathbb E_{s'\sim d}[Z(s')]} \ell_s(\theta) \right]. \end{aligned} }$$ FERPO therefore preserves the rollout state weighting, while batch-wide normalization gives greater weight to states with larger $Z(s)$. This difference remains even with unlimited samples. We explain this normalization difference and the additional components of full V-MPO in more detail in Appendix A.6.1 (lines 2112–2165). **Additional comparisons.** We evaluated MORE-inspired and V-MPO variants on four DMC tasks at approximately 50M interactions, alongside FERPO and REPPO’s main-benchmark results. The MORE-inspired adaptation optimizes the entropy-regularized actor objective with an always-active reverse-KL trust-region penalty. For a fair comparison, the MORE and V-MPO variants use matched network sizes, learning rates, rollout settings, and optimization budgets (Appendix A.6.2). The final evaluation returns are IQMs across seeds, including all four completed V-MPO seeds per task: | Task | FERPO | REPPO | MORE-inspired | V-MPO variant | |---|---:|---:|---:|---:| | AcrobotSwingup | **500.9** | 345.5 | 366.0 | 304.9 | | AcrobotSwingupSparse | **36.3** | 7.6 | 5.0 | 0.5 | | FingerSpin | **736.8** | 649.9 | 575.1 | 711.7 | | FishSwim | **724.3** | 643.4 | 127.0 | 137.1 | FERPO achieves the highest reported final return on all four tasks. REPPO outperforms the MORE-inspired variant on three of four tasks, supporting the performance benefits of REPPO’s design choices in this comparison. Appendix A.6.2 presents learning curves (Figure 20) and final returns (Table 7); Section 5.3 summarizes the results. **References** [1] Abdolmaleki et al. “Model-Based Relative Entropy Stochastic Search.” NeurIPS, 2015. [2] Song et al. “V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control.” ICLR, 2020.
AuthorsOpenReview ↗

Answer to Review 2/3

W2: Evidence for exploration

With rollout-policy sampling, the preferred direction is recovered in only 26.4% of batches even at K = 1,024. It remains unclear whether the practical small-K algorithm achieves the claimed exploration benefit.

A2: The initialization at μ0=−0.6\mu_0=-0.6 in Section 5.1 makes the better well rarely sampled. In the initial-mean experiment in Appendix A.1.1 (Table 2), we test how learning with K=16K=16 rollout-policy samples changes when the initial mean is moved closer to that well. At every tested initialization, REPPO's exact initial update points toward the worse basin, whereas FERPO's points toward the better basin.

Both methods use 16 fresh rollout-policy samples per update, with 80 mean updates and 2,000 paired seeds per initialization. We count a run as reaching the better basin when its final policy mean lies to the right of −0.04829-0.04829. This is the peak between the two minima of the expected cost and marks the boundary between their basins (Appendix A.1.1, Table 2).

Initial meanFERPO: better-basin selectionREPPO: better-basin selection
-0.60.70%0.00%
-0.43.80%0.00%
-0.222.95%0.00%
-0.144.40%3.80%

FERPO selects the better basin more often at every tested initialization. Moving the initial mean from −0.6-0.6 to −0.1-0.1 increases its selection rate from 0.7% to 44.4%, compared with 0% to 3.8% for REPPO.

Exploration in an RL setting with adaptive ESS. We test exploration in an eight-state Markov decision process (MDP), where actions determine both reward and the next state. The states represent eight equally spaced headings on a circle, with one-dimensional actions a∈[−1,1]a\in[-1,1]. At each state, the reward follows the asymmetric double-well landscape from Section 5.1: one action well gives higher reward than the other. Its location is independently randomized to the left or right in each state and remains fixed during training. The policy must therefore learn which well is better at each state.

The barrier parameter BB controls the difficulty: increasing it raises the cost of moving between the wells. We test B∈0.25,1,2,3,4,6,8,12B\in\\{0.25,1,2,3,4,6,8,12\\} to examine whether FERPO’s practical sampler can discover the better wells as exploration becomes harder.

Both methods start near the inferior well with initial latent standard deviation 0.40.4, use fixed temperature 0.050.05, and receive 204,800 environment interactions per run. FERPO uses K=16K=16 candidates from the rollout policy, adaptive ESS control, and no proposal widening.

We compare a learned 64×6464\times64 scalar MSE critic with an oracle QπQ^\pi critic computed from the known MDP without fitting error (see Appendix A.1.2 for more details).

The table reports the mean number of states, out of eight, whose final policy assigns at least 80% probability to the better basin, over 36 paired seeds per setting.

With the learned critic, FERPO's advantage is statistically significant at every tested B≥3B\geq3, after correction for multiple comparisons.

Barrier BBFERPO, oracleREPPO, oracleFERPO, learnedREPPO, learned
0.257.947.177.977.94
15.860.006.316.44
24.060.004.313.83
33.640.004.532.72
43.560.004.692.03
63.220.004.062.11
82.080.004.501.89
120.890.004.171.97

At B=4B=4, FERPO favors the better basin in 4.69 states versus 2.03 for REPPO with the learned critic; at B=12B=12, the counts remain 4.17 versus 1.97. This provides evidence of an exploration benefit in an RL setting across difficult landscapes, using FERPO’s practical sampler, learned critic, and adaptive ESS control.

With the oracle critic, REPPO fails to favor the better basin in any state at every tested B≥1B\geq1, consistent with becoming trapped near the inferior well. A possible explanation is that accurate local gradients keep its updates within that well, while errors in a learned critic can perturb updates and encourage exploration. FERPO also selects better basins with the oracle critic, although its basin count decreases at the highest barriers.

Figure 2 in Section 5.1 shows how exploration changes with the barrier height. Further details of the environment and experiments are given in Appendix A.1.2.

AuthorsOpenReview ↗

Answer to Review 3/3

W2: Robustness to critic-gradient errors

The toy example uses an exact critic and therefore does not test robustness to critic-gradient errors.

A3: We added a controlled experiment with an imperfect critic to address this concern directly. We consider minimizing the quadratic cost c(a)=(a−0.5)2c(a)=(a-0.5)^2 over a∈(−1,1)a\in(-1,1), whose minimum is at a=0.5a=0.5. We implement this as a continuous-action bandit with reward Q(a)=−c(a)Q(a)=-c(a). Both methods receive

Q^n(a)=Q(a)+n−1cos⁡(n2a).\widehat Q_n(a)=Q(a)+n^{-1}\cos(n^2a).

As nn increases, the value error approaches zero while the critic action-gradient error grows without bound: ∣Q^n−Q∣∞≤1/n→0\\|\widehat Q_n-Q\\|_\infty\leq1/n\to0, but ∣∇aQ^n−∇aQ∣∞=n→∞\\|\nabla_a\widehat Q_n-\nabla_aQ\\|_\infty=n\to\infty.

We use 20 paired seeds, 512 updates, and 64 critic queries per update. Both methods learn the mean and standard deviation of a tanh-Gaussian policy. The table shows mean final reward for every tested nn; higher is better.

CriticFERPO rewardREPPO reward
Exact QQ-0.049-0.045
n=8n=8-0.048-0.046
n=16n=16-0.049-0.047
n=32n=32-0.049-0.072
n=48n=48-0.049-0.081
n=64n=64-0.049-0.141
n=96n=96-0.049-0.216
n=128n=128-0.049-0.349
n=256n=256-0.049-0.414

FERPO maintains nearly the same learning speed and final reward as with the exact critic, whereas REPPO becomes slower and reaches worse rewards (Figure 21(c), Table 8). Figure 21(a) shows the corresponding gradient errors: FERPO's estimate becomes more accurate as the value error shrinks, while REPPO's becomes noisier.

Appendix D proves that, for fixed actions, actor, and multipliers, FERPO's gradient perturbation vanishes as O(1/n)O(1/n). REPPO's sampled-gradient error instead has standard deviation proportional to n/Nn/\sqrt N, growing without bound at fixed sample count NN. The growing variance makes finite-sample updates increasingly noisy.

Q2: Sampled-update accuracy

With the practical candidate budget and adaptive ESS control, how closely do sampled updates match a high-sample reference?

A4: With small KK, the gradient estimate at an individual state can be inaccurate. In practice, however, actor updates average gradients over many different states, which can reduce sampling noise. We test how this improves agreement with a high-sample forward-KL reference on dense AcrobotSwingup. The reference uses 16,384–131,072 actions per state, with the actor, critic, rollout policy, and multipliers fixed.

Across three training seeds, at K=16K=16 and 29.88M interactions, increasing the state batch from 8 to 2,048 raises cosine similarity from 0.749 to 0.981 and reduces relative RMSE from 0.844 to 0.185. Thus, averaging over states substantially improves the small-KK estimate on this task, although finite-sample bias can remain. Appendix A.3.10, Figure 19, and Table 6 give the analysis and results across five training checkpoints.

We thank the reviewer for helping us develop a clearer comparison with MORE and V-MPO and stronger tests of the practical sampled update. Their questions led us to add the controlled RL exploration experiment, the critic-gradient stability analysis and experiments, and the study of gradient accuracy when averaging over states. These additions clarify both the evidence for FERPO’s benefits and the limits of its finite-sample implementation.

References refer to the updated manuscript unless explicitly identified as figures from the original submission. Manuscript revisions are highlighted in blue.

Thanks for your reply and more experiment results. My main concerns are mostly addressed. Raised my score to 3.

Reviewer_XPHoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an on-policy reinforcement learning method that fits an actor to an entropy- and KL-regularized target without using critic action gradients. It uses self-normalized importance sampling and controls weight concentration through effective sample size. Experiments on 34 control and manipulation tasks report competitive performance and lower update costs, with additional studies of cached updates and a state-value variant.

Critical strengths
  1. The actor update provides a practical alternative to differentiating through the critic. The target construction, per-state weight normalization, and ESS controller form a coherent algorithm. The analysis also distinguishes population guarantees from the weaker guarantees available with sampled updates.

  2. The evaluation demonstrates a useful performance–cost trade-off across several task families. FERPO achieves competitive final results on DMC and ManiSkill, while the matched timing study reports lower optimization cost. The candidate-count, entropy, and caching ablations help explain the practical design choices.

Critical weaknesses
  1. The exploration argument is not fully established for the implemented sampler. In Section 5.1, even 1,024 rollout-policy samples recover the preferred update direction in only 26.4% of batches; much better recovery requires a wider proposal. The practical algorithm samples from the rollout policy. Controlling empirical ESS cannot reveal important modes that were never sampled, so the connection between the exact forward-KL argument and the observed learning gains remains uncertain.

  2. The benchmark comparisons are not fully standardized. Appendix A.3.7 combines archived baseline runs with FERPO evaluation exports that use different evaluation sample counts and changing seed cohorts. The confidence intervals are conditional on the available runs and selected configurations. These limitations matter when final-performance differences are small. The main paper should summarize the evaluation differences and explain why some runs lack evaluation records.

Questions for authors

Please address the concerns listed in the Weaknesses section above.

3 comments
AuthorsOpenReview ↗

Answer to Review 1/2

We thank the reviewer for their careful reading and constructive comments. We appreciate their recognition of the coherent actor-update construction, the distinction between population and sampled guarantees, and the useful performance–cost trade-off across benchmarks. We also appreciate their positive assessment of the design ablations. Their concerns focus on the exploration argument under rollout-policy sampling and the comparability of benchmark evaluations. We address these points below, including the limitations that remain.

W1: Exploration with the implemented rollout-policy sampler

Does FERPO's exploration benefit persist with the implemented rollout-policy sampler, despite limited coverage of the better well and the inability of ESS control to detect unsampled modes?

A1: The initialization at μ0=−0.6\mu_0=-0.6 in Section 5.1 makes the better well rarely sampled. We added an experiment with four initial policy means, μ0∈−0.6,−0.4,−0.2,−0.1\mu_0\in\\{-0.6,-0.4,-0.2,-0.1\\}, moving the starting policy progressively closer to the better well. We test whether this improves the chance of reaching that well with K=16K=16 rollout-policy samples per update (Appendix A.1.1, Table 2). At these four starting means, FERPO’s update computed with exact expectations points toward the better basin, while REPPO’s points toward the worse basin. The sampled FERPO updates used in the experiment can point in either direction; the table measures how often repeated updates actually reach the better basin.

Both methods use 16 fresh rollout-policy samples per update, with 80 mean updates and 2,000 paired seeds per initialization. We count a run when its final mean exceeds −0.04829-0.04829, the local maximum separating the two expected-cost minima; means to its right lie in the better basin (Appendix A.1.1, Table 2).

Initial meanFERPO: better-basin selectionREPPO: better-basin selection
-0.60.70%0.00%
-0.43.80%0.00%
-0.222.95%0.00%
-0.144.40%3.80%

FERPO selects the better basin more often at every tested initialization. Moving the initial mean from −0.6-0.6 to −0.1-0.1 increases its selection rate from 0.7% to 44.4%, compared with 0% to 3.8% for REPPO.

New convergence result. Proposition 5 in Appendix C gives a complementary theoretical explanation. For a related class of symmetric polynomial costs, exact fixed-variance Gaussian updates converge globally under FERPO, while REPPO can remain trapped at suboptimal minima, under the stated conditions.

Exploration in an RL setting with adaptive ESS. We test repeated updates with the implemented rollout-policy sampler in an eight-state Markov decision process (MDP), where actions determine both reward and the next state.

We added a sequential control task with eight headings arranged on a circle and one-dimensional actions in [−1,1][-1,1] and asymmetric double-well rewards based on Section 5.1. One well gives higher reward; its left/right location is independently randomized across states and fixed during training. We increase the central cost barrier through B∈0.25,1,2,3,4,6,8,12B\in\\{0.25,1,2,3,4,6,8,12\\}. Both methods start near the inferior well with initial standard deviation 0.40.4, use temperature 0.050.05, and receive 204,800 interactions. FERPO uses K=16K=16 rollout-policy candidates and adaptive ESS control.

We compare a learned 64×6464\times64 scalar MSE critic against an oracle QπQ^\pi critic evaluated directly from the known model without fitting error.

The table reports the mean number of states, out of eight, whose final policy assigns at least 80% probability to the better basin, over 36 paired seeds per setting. With the learned critic, FERPO's advantage is statistically significant at every tested B≥3B\geq3, after correction for multiple comparisons.

Barrier BBFERPO, oracleREPPO, oracleFERPO, learnedREPPO, learned
0.257.947.177.977.94
15.860.006.316.44
24.060.004.313.83
33.640.004.532.72
43.560.004.692.03
63.220.004.062.11
82.080.004.501.89
120.890.004.171.97

With the learned critic, FERPO favors the better basin in more than twice as many states as REPPO at B=4,8,12B=4,8,12. Thus, the learning advantage occurs with the practical K=16K=16 rollout-policy sampler across multiple hard settings. This result concerns repeated learning updates and remains compatible with the limitation of an individual SNIS estimate when a mode is absent from its samples.

Figure 2 in Section 5.1 compares the methods across barrier heights. See Appendix A.1.2 for details of the environment and experiments.

AuthorsOpenReview ↗

Answer to Review 2/2

Additional evidence: averaging over states. Averaging gradients over many states can reduce the sampling noise in a small-KK update. On dense AcrobotSwingup, at K=16K=16 and 29.88M interactions, increasing the state batch from 8 to 2,048 raises cosine similarity with a high-sample forward-KL reference from 0.749 to 0.981. See Figure 19 and Table 6 in Appendix A.3.10 for results at the other training checkpoints and state batch sizes.

W2: Benchmark standardization and interpretation of uncertainty

Please clarify differences in evaluation protocols, missing evaluation records, and how these affect the interpretation of confidence intervals.

A2: We completed the six missing Walker evaluations for seeds 11–12 and updated the benchmark plots and comparisons. FERPO's DMC IQM changes only from 897.655 to 897.712, leaving the endpoint-comparison conclusions unchanged (Figures 5 and 16; Appendix A.3.7, Table 5). Each environment retains equal weight. We now specify the evaluation episode count for the completed Walker reruns in the main paper. These evaluations use 1,024 episodes per checkpoint, compared with 128 for the original runs.

Baseline configurations are fixed across tasks within each benchmark, with no selection of individual seeds by performance. Appendix A.3.6 describes the data sources, evaluation procedures, and baseline selection.

We thank the reviewer again for their feedback. Their comments have helped us sharpen the scope of the exploration claims and improve the transparency of the evaluation protocols and uncertainty estimates.

References refer to the updated manuscript unless explicitly identified as figures from the original submission. Manuscript revisions are highlighted in blue.

Thank you for the additional experiments and clarifications. My main concerns have been substantially addressed. My positive rating of 3 will be maintained.

Reviewer_ime7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum-entropy reinforcement learning algorithm that performs policy improvement using critic values. The method derives a target action distribution through entropy and KL regularization and updates the actor by minimizing the forward KL divergence to this target. Experiments on MuJoCo Playground and ManiSkill evaluate the proposed method against existing approaches, including Relative Entropy Pathwise Policy Optimization (REPPO), with additional computational benchmarks examining actor update efficiency.

Critical strengths
  1. The proposed approach of constructing a regularized target action distribution using critic values and updating the actor through forward KL divergence provides an interesting alternative to critic-gradient-based policy optimization.
  2. The experimental evaluation includes multiple continuous-control environments and computational benchmarks, providing some evidence of the proposed method's performance and actor update efficiency.
Critical weaknesses
  1. The novelty and theoretical justification of the proposed algorithm are limited. Although FERPO introduces a different actor update mechanism, the paper does not provide substantial theoretical results, such as performance bounds, to establish its advantages over existing policy optimization approaches. It remains unclear under what conditions the proposed formulation would be preferable to existing methods and what specific benefits arise from this modification.
  2. The empirical results do not sufficiently support the claimed advantages of the proposed method. In Figure 4, FERPO and REPPO appear to achieve comparable performance, with only a marginal advantage during early training. While the paper reports faster actor updates, it is unclear whether computational efficiency is the primary contribution or whether the method is expected to improve overall learning performance. Furthermore, the interpretation of Figure 2 is unclear, including whether higher or lower values indicate better performance and how the reported results relate to the main claims. Overall, the experiments provide limited evidence of meaningful improvements over the corresponding baseline.
Questions for authors
  1. What theoretical properties or performance guarantees justify the proposed actor update mechanism, and under what conditions should it outperform existing approaches?
  2. Given the similar performance of FERPO and REPPO in Figure 4, what is the primary advantage of the proposed method beyond faster actor updates?
  3. What does Figure 2 demonstrate, and how should the reported values be interpreted in relation to policy performance?
  4. How do the reported computational improvements translate into overall training efficiency and sample efficiency compared with REPPO?
3 comments
AuthorsOpenReview ↗

Answer to Rebuttal 1/2

We thank the reviewer for their careful assessment and detailed questions. We appreciate their recognition that fitting a regularized target through forward KL provides an interesting alternative to critic-gradient-based policy optimization, and that the evaluation provides evidence of performance and actor-update efficiency. Their feedback highlights the need to explain the contribution, theoretical scope, and practical benefits more precisely. Below, we address these points and clarify how the computational measurements relate to learning performance.

W1 / Q1: Novelty, theoretical justification, and conditions for an advantage

Please clarify FERPO's contribution, theoretical guarantees, and the conditions under which its actor update offers an advantage.

A1:

New result 1: Convergence on multimodal cost landscapes. We have added a convergence analysis for minimizing a cost function under Gaussian policies with fixed variance (Proposition 5, Appendix C). For the specified class of symmetric polynomial costs, sufficiently small policy variance and KL regularization ensure that FERPO converges geometrically to the globally optimal mean from every initialization when its step size satisfies 0<hF≤σ20<h_{\mathrm F}\leq\sigma^2. REPPO can instead get stuck at a suboptimal stationary mean, where its expected-cost gradient is zero. With a sufficiently small step size, nearby initializations also converge to that suboptimal minimum. The result extends to separable costs in arbitrary dimension with arbitrarily many local minima.

New result 2: Stability under critic errors. We have also added an analysis of sensitivity to critic errors (Appendix D). For the perturbation

\nabla_a e_n(a)=-n\sin(n^2a),$$ we have $\\|e_n\\|_\infty\leq1/n\to0$ but $\\|\nabla_a e_n\\|_\infty=n\to\infty$ as $n\to\infty$: values become accurate while their action derivatives become increasingly inaccurate. For any fixed candidate batch, actor, and multipliers, FERPO's gradient perturbation is bounded by $G/[n(\tau+\lambda)]\to0$, where $G$ is the largest policy log-density gradient norm in that batch. Using the same sampled actions, FERPO’s gradients computed with the perturbed and exact critics become arbitrarily close as $n$ increases. This holds for any fixed number of actions, even as the perturbed critic’s action derivatives grow without bound in magnitude. For REPPO, the variance of this gradient error instead grows proportionally to $n^2/N$ for $N$ independent actions, corresponding to a standard deviation proportional to $n/\sqrt N$. Its noise therefore diverges at fixed sample count. This does not imply divergence of the expected gradient. The experiments in A2 test how this gradient stability affects learning. **Existing result: ESS and policy improvement.** The original submission already connected population ESS to policy total variation through Proposition 1 and to the policy-improvement bound in Section 2, Equation (3). **W2 / Q4: Evidence of improvement, computational efficiency, and sample efficiency** Please clarify FERPO's learning benefits and explain how lower update costs translate into overall training efficiency, including the interpretation of the computational results. **A2:** To assess overall training efficiency, we measure environment collection and actor–critic optimization for the tasks used in our benchmarks, using the same A100 GPU for both methods within each task. The resulting seconds per interaction convert the existing learning curves into estimated training time. These results are now shown in Figure 9. For each benchmark, we compare both methods over the longest estimated training-time interval covered by all included runs, without extrapolating beyond their recorded evaluations. The table compares task-balanced IQMs at its endpoint, giving both methods the same estimated time budget. Brackets show 95% seed-bootstrap confidence intervals; they do not include uncertainty in the timing measurements. | Benchmark | Metric | Time (min) | FERPO | REPPO | |---|---|---:|---:|---:| | DMC | Return | 35.6 | **892.0** [884.4, 899.0] | 869.1 [861.5, 876.4] | | G1/T1 | Return | 25.3 | 31.40 [31.01, 31.79] | **33.92** [33.72, 34.13] | | ManiSkill | Success (%) | 71.3 | **98.73** [97.02, 99.55] | 94.31 [90.93, 97.14] | FERPO achieves higher performance at the compared estimated time budgets on DMC and ManiSkill, while REPPO remains ahead on G1/T1. See Appendix A.2.4 for more details. Together with the interaction-based results, this suggests that FERPO can reach better performance sooner both in environment interactions and in estimated wall-clock time.
AuthorsOpenReview ↗

Answer to Review 2/2

A2 (continued): Critic-gradient robustness and learning speed. The analysis in A1 identifies a mechanism that can slow REPPO’s learning: inaccurate critic derivatives increase the variance of its sampled actor gradient, making updates noisier. FERPO avoids this source of noise by using critic values rather than their action derivatives. We test this in a continuous-action bandit with reward Q(a)=−(a−0.5)2Q(a)=-(a-0.5)^2, using the perturbed critics Q^n=Q+en\widehat Q_n=Q+e_n from A1. Both methods receive 64 critic queries per update and train for 512 updates over 20 paired seeds. The table reports mean final reward; higher is better.

CriticFERPO rewardREPPO reward
Exact QQ-0.049-0.045
n=8n=8-0.048-0.046
n=16n=16-0.049-0.047
n=32n=32-0.049-0.072
n=48n=48-0.049-0.081
n=64n=64-0.049-0.141
n=96n=96-0.049-0.216
n=128n=128-0.049-0.349
n=256n=256-0.049-0.414

FERPO maintains nearly its exact-critic learning speed and final reward, whereas REPPO becomes slower and reaches worse rewards as the critic derivatives deteriorate. Figure 21(a) shows the growing gradient error in REPPO, and Figure 21(c) shows the resulting slowdown in learning; Appendix D and Table 8 provide the full results. Because the update and sampling budgets are equal, this experiment links critic-gradient robustness to learning progress, complementing the computational savings measured above.

Q2: Primary advantage beyond faster actor updates

Given the similar FERPO and REPPO performance in Figure 4 of the original submission, what is the primary advantage beyond faster actor updates?

A3: We have added a controlled RL experiment to test whether FERPO explores better action regions more effectively than REPPO.

The experiment is an eight-state Markov decision process (MDP), with headings arranged on a circle, one-dimensional bounded actions, and asymmetric double-well rewards adapted from Section 5.1. The better well's left/right location is randomized independently across states, and increasing BB raises the central cost barrier. Both methods start near the inferior well with initial latent std 0.40.4, use fixed temperature 0.050.05, and receive the same 204,800 interactions. FERPO uses K=16K=16 rollout-policy candidates, adaptive ESS, and no proposal widening. We compare a learned 64×6464\times64 MSE critic and an oracle QπQ^\pi critic computed from the known model without fitting error.

The table reports the mean number of states, out of eight, whose final policy assigns at least 80% probability to the better basin, over 36 paired seeds per setting. With the learned critic, FERPO favors the better basin in more states at every tested high barrier, B∈3,4,6,8,12B\in\\{3,4,6,8,12\\}, with statistically significant differences after correction for multiple comparisons.

Barrier BBFERPO, oracleREPPO, oracleFERPO, learnedREPPO, learned
0.257.947.177.977.94
15.860.006.316.44
24.060.004.313.83
33.640.004.532.72
43.560.004.692.03
63.220.004.062.11
82.080.004.501.89
120.890.004.171.97

For example, with the learned critic at B=12B=12, FERPO favors the better basin in 4.17 states versus 1.97 for REPPO. These results provide controlled evidence of better exploration in an RL setting: FERPO learns to favor the better basin in more states at the same interaction budget.

Figure 2 in Section 5.1 compares the methods across barrier heights. See Appendix A.1.2 for details of the environment and experiments.

Q3: Meaning and interpretation of Figure 2

What does Figure 2 of the original submission demonstrate, and how should its reported values be interpreted in relation to policy performance?

A4: Figure 2 of the original submission (Figure 3 in the updated manuscript) compares actor-update and combined actor–critic update times. Lower is better. FERPO can perform faster updates, helping it reach better performance sooner.

We thank the reviewer again for their constructive feedback. Their questions have helped us clarify the contribution and theoretical scope, improve the explanation of the computational results, and distinguish sample efficiency from wall-clock training efficiency.

References refer to the updated manuscript unless explicitly identified as figures from the original submission. Manuscript revisions are highlighted in blue.

AuthorsOpenReview ↗

Follow-up on our response to your review”

Dear Reviewer ime7,

Thank you again for your review. We added theoretical analyses of convergence and sensitivity to critic errors, controlled RL experiments investigating exploration, and comparisons at equal estimated training-time budgets. We also clarified the interpretation of the computational figures and how they relate to learning performance.

As the discussion period nears its end, we would appreciate your feedback on whether these results address your concerns and whether any points need further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and discussion

We thank the reviewers for their constructive feedback. Reviewer XPHo confirmed that “My main concerns have been substantially addressed,” maintaining their positive recommendation. Reviewer 96Y9 stated that “My main concerns are mostly addressed” and raised their recommendation to weak acceptance.

We submitted our rebuttal more than five days before the discussion deadline and subsequently posted a follow-up reminder. At the time of posting this summary, about one hour before the deadline, we have not received a response from Reviewer ime7. Below, we summarize the reviewers’ main concerns and how we addressed them.

1. Global convergence on multimodal problems. For a class of multimodal costs with fixed-variance Gaussian policies, we prove global convergence of our method under the stated conditions, while REPPO can converge to suboptimal local minima. This formalizes an advantage consistent with our controlled multimodal optimization experiments.

See our response to ime7, A1; Proposition 5, Appendix C.

2. Robustness to inaccurate critic gradients. We analyze and test perturbed critics

Q^n(a)=Q(a)+cos⁡(n2a)n,\widehat Q_n(a)=Q(a)+\frac{\cos(n^2a)}{n},

for which

\qquad \lVert \nabla_a\widehat Q_n-\nabla_a Q\rVert_\infty=n\to\infty.$$ We prove that, for fixed policy parameters, sampled actions, and multipliers, FERPO’s policy gradient computed with $\widehat Q_n$ approaches the gradient computed with the exact critic $Q$ as $n$ grows. REPPO’s gradient error instead has growing variance at a fixed sample count. We verify these predictions empirically: FERPO’s measured gradient error decreases, while REPPO’s becomes noisier. Learning experiments show the corresponding performance effect: as $n$ grows, FERPO retains nearly the learning speed and final reward obtained with the exact critic, whereas REPPO learns more slowly and reaches lower rewards. See our response to ime7, A1–A2, and to 96Y9, A3; Appendix D. **3. Better exploration with practical sampling.** To address whether the exploration benefit persists with the implemented sampler and small candidate budgets, we test the multimodal optimization problem with $K=16$ at different initial policy means. FERPO reaches the better well more often at every tested initialization; at $\mu_0=-0.1$, it does so in 44.4% of runs, versus 3.8% for REPPO. We also construct an eight-state RL problem requiring exploration between optimal and suboptimal action basins at each state. With matched interaction budgets, FERPO learns to favor the better basin in more states across difficult settings, both when the critic is learned and when it is computed exactly. These results demonstrate better exploration with practical sampling, beyond faster actor updates. See our response to ime7, A3, to XPHo, A1, and to 96Y9, A2; Appendix A.1. **4. Learning performance at equal estimated training time.** We measured environment collection and actor–critic optimization on the same A100 GPU and used these costs to express learning curves against estimated training time. At equal estimated time budgets, FERPO achieves higher performance on DMC and ManiSkill, while REPPO remains ahead on G1/T1. These comparisons show that cheaper updates can translate into better performance within the same estimated training time. See our response to ime7, A2; Figure 9 and Appendix A.2.4. **5. Clearer positioning and additional baselines.** We expanded the comparison with MORE and MPO/V-MPO, explaining differences in KL direction, state weighting, critic learning, and regularization. We also evaluated MORE-inspired and V-MPO variants under fair conditions. FERPO achieves the highest final return among the compared methods on all four evaluated DMC tasks, supporting the practical value of its design choices. See our response to 96Y9, A1; Appendix A.6. **6. Accuracy of the practical sampled update.** On AcrobotSwingup with 16 candidates per state, increasing the state batch from 8 to 2,048 raises gradient cosine similarity with a high-sample reference from 0.749 to 0.981. This shows that averaging across states can substantially improve the practical gradient estimate. See our response to 96Y9, A4, and to XPHo; Appendix A.3.10. We addressed Reviewer ime7’s concerns about theoretical justification and practical learning benefits through new convergence and robustness analyses, exploration experiments, and comparisons at equal estimated training time (see points 1–4 above and our response to ime7, A1–A3). Without a response to our rebuttal or follow-up, we could not determine whether any concerns remained or provide further clarification. All revisions are incorporated into the updated manuscript and highlighted in blue. We thank all reviewers for their careful assessments and constructive suggestions, which have substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.