- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FERPO is an on-policy maximum-entropy RL method that constructs a regularized target action distribution and fits the actor through forward KL. It uses self-normalized importance sampling and adaptive ESS control, avoiding critic action gradients. Experiments on MuJoCo Playground and ManiSkill show competitive performance, sample-efficiency gains on some tasks, and reduced update costs.
Critical strengths
The method provides a practical actor update using critic values without requiring action derivatives. The entropy and ESS controls are well explained, and the limitations of the theoretical guarantees are clearly stated. The evaluation covers 34 tasks, with useful ablations and runtime measurements. Competitive results and lower update costs suggest practical value for parallel on-policy learning.
Critical weaknesses
The contribution needs clearer positioning against closely related work. MORE already derives the same entropy- and KL-constrained target form, while MPO uses forward-KL policy fitting. MPO is discussed, but MORE and the on-policy V-MPO method are missing. The soft critic and ESS controller provide differences, but their added value over existing policy-fitting methods is not sufficiently established by direct comparisons. Evidence for the main motivation remains limited. The toy example uses an exact critic, so it does not test robustness to critic-gradient errors. With rollout-policy sampling, it recovers the preferred direction in only 26.4% of batches even at K = 1,024. Although this limitation is acknowledged, it remains unclear whether the practical small-K algorithm achieves the proposed exploration benefit.
Questions for authors
-
Could you clarify the contribution relative to MORE and MPO/V-MPO, and provide a matched comparison that isolates the benefit of FERPO’s additional design choices?
-
With the practical candidate budget and adaptive ESS control, how closely do sampled updates match a high-sample reference? What evidence connects the benchmark gains to improved exploration or reduced sensitivity to critic-gradient errors?
4 comments
Answer to Review 2/3
W2: Evidence for exploration
With rollout-policy sampling, the preferred direction is recovered in only 26.4% of batches even at K = 1,024. It remains unclear whether the practical small-K algorithm achieves the claimed exploration benefit.
A2: The initialization at in Section 5.1 makes the better well rarely sampled. In the initial-mean experiment in Appendix A.1.1 (Table 2), we test how learning with rollout-policy samples changes when the initial mean is moved closer to that well. At every tested initialization, REPPO's exact initial update points toward the worse basin, whereas FERPO's points toward the better basin.
Both methods use 16 fresh rollout-policy samples per update, with 80 mean updates and 2,000 paired seeds per initialization. We count a run as reaching the better basin when its final policy mean lies to the right of . This is the peak between the two minima of the expected cost and marks the boundary between their basins (Appendix A.1.1, Table 2).
| Initial mean | FERPO: better-basin selection | REPPO: better-basin selection |
|---|---|---|
| -0.6 | 0.70% | 0.00% |
| -0.4 | 3.80% | 0.00% |
| -0.2 | 22.95% | 0.00% |
| -0.1 | 44.40% | 3.80% |
FERPO selects the better basin more often at every tested initialization. Moving the initial mean from to increases its selection rate from 0.7% to 44.4%, compared with 0% to 3.8% for REPPO.
Exploration in an RL setting with adaptive ESS. We test exploration in an eight-state Markov decision process (MDP), where actions determine both reward and the next state. The states represent eight equally spaced headings on a circle, with one-dimensional actions . At each state, the reward follows the asymmetric double-well landscape from Section 5.1: one action well gives higher reward than the other. Its location is independently randomized to the left or right in each state and remains fixed during training. The policy must therefore learn which well is better at each state.
The barrier parameter controls the difficulty: increasing it raises the cost of moving between the wells. We test to examine whether FERPO’s practical sampler can discover the better wells as exploration becomes harder.
Both methods start near the inferior well with initial latent standard deviation , use fixed temperature , and receive 204,800 environment interactions per run. FERPO uses candidates from the rollout policy, adaptive ESS control, and no proposal widening.
We compare a learned scalar MSE critic with an oracle critic computed from the known MDP without fitting error (see Appendix A.1.2 for more details).
The table reports the mean number of states, out of eight, whose final policy assigns at least 80% probability to the better basin, over 36 paired seeds per setting.
With the learned critic, FERPO's advantage is statistically significant at every tested , after correction for multiple comparisons.
| Barrier | FERPO, oracle | REPPO, oracle | FERPO, learned | REPPO, learned |
|---|---|---|---|---|
| 0.25 | 7.94 | 7.17 | 7.97 | 7.94 |
| 1 | 5.86 | 0.00 | 6.31 | 6.44 |
| 2 | 4.06 | 0.00 | 4.31 | 3.83 |
| 3 | 3.64 | 0.00 | 4.53 | 2.72 |
| 4 | 3.56 | 0.00 | 4.69 | 2.03 |
| 6 | 3.22 | 0.00 | 4.06 | 2.11 |
| 8 | 2.08 | 0.00 | 4.50 | 1.89 |
| 12 | 0.89 | 0.00 | 4.17 | 1.97 |
At , FERPO favors the better basin in 4.69 states versus 2.03 for REPPO with the learned critic; at , the counts remain 4.17 versus 1.97. This provides evidence of an exploration benefit in an RL setting across difficult landscapes, using FERPO’s practical sampler, learned critic, and adaptive ESS control.
With the oracle critic, REPPO fails to favor the better basin in any state at every tested , consistent with becoming trapped near the inferior well. A possible explanation is that accurate local gradients keep its updates within that well, while errors in a learned critic can perturb updates and encourage exploration. FERPO also selects better basins with the oracle critic, although its basin count decreases at the highest barriers.
Figure 2 in Section 5.1 shows how exploration changes with the barrier height. Further details of the environment and experiments are given in Appendix A.1.2.
Answer to Review 3/3
W2: Robustness to critic-gradient errors
The toy example uses an exact critic and therefore does not test robustness to critic-gradient errors.
A3: We added a controlled experiment with an imperfect critic to address this concern directly. We consider minimizing the quadratic cost over , whose minimum is at . We implement this as a continuous-action bandit with reward . Both methods receive
As increases, the value error approaches zero while the critic action-gradient error grows without bound: , but .
We use 20 paired seeds, 512 updates, and 64 critic queries per update. Both methods learn the mean and standard deviation of a tanh-Gaussian policy. The table shows mean final reward for every tested ; higher is better.
| Critic | FERPO reward | REPPO reward |
|---|---|---|
| Exact | -0.049 | -0.045 |
| -0.048 | -0.046 | |
| -0.049 | -0.047 | |
| -0.049 | -0.072 | |
| -0.049 | -0.081 | |
| -0.049 | -0.141 | |
| -0.049 | -0.216 | |
| -0.049 | -0.349 | |
| -0.049 | -0.414 |
FERPO maintains nearly the same learning speed and final reward as with the exact critic, whereas REPPO becomes slower and reaches worse rewards (Figure 21(c), Table 8). Figure 21(a) shows the corresponding gradient errors: FERPO's estimate becomes more accurate as the value error shrinks, while REPPO's becomes noisier.
Appendix D proves that, for fixed actions, actor, and multipliers, FERPO's gradient perturbation vanishes as . REPPO's sampled-gradient error instead has standard deviation proportional to , growing without bound at fixed sample count . The growing variance makes finite-sample updates increasingly noisy.
Q2: Sampled-update accuracy
With the practical candidate budget and adaptive ESS control, how closely do sampled updates match a high-sample reference?
A4: With small , the gradient estimate at an individual state can be inaccurate. In practice, however, actor updates average gradients over many different states, which can reduce sampling noise. We test how this improves agreement with a high-sample forward-KL reference on dense AcrobotSwingup. The reference uses 16,384–131,072 actions per state, with the actor, critic, rollout policy, and multipliers fixed.
Across three training seeds, at and 29.88M interactions, increasing the state batch from 8 to 2,048 raises cosine similarity from 0.749 to 0.981 and reduces relative RMSE from 0.844 to 0.185. Thus, averaging over states substantially improves the small- estimate on this task, although finite-sample bias can remain. Appendix A.3.10, Figure 19, and Table 6 give the analysis and results across five training checkpoints.
We thank the reviewer for helping us develop a clearer comparison with MORE and V-MPO and stronger tests of the practical sampled update. Their questions led us to add the controlled RL exploration experiment, the critic-gradient stability analysis and experiments, and the study of gradient accuracy when averaging over states. These additions clarify both the evidence for FERPO’s benefits and the limits of its finite-sample implementation.
References refer to the updated manuscript unless explicitly identified as figures from the original submission. Manuscript revisions are highlighted in blue.
Thanks for your reply and more experiment results. My main concerns are mostly addressed. Raised my score to 3.
Answer to Review 1/3
We thank the reviewer for their constructive feedback on the contribution, comparisons, and sampled updates.
W1 / Q1: Contribution and comparisons with MORE, MPO, and V-MPO
Clarify FERPO’s contribution relative to MORE and MPO/V-MPO, and provide comparisons supporting its design choices.
A1: We now discuss MORE [1] and V-MPO [2] in Section 4 (lines 260–277) and Appendix A.6 (lines 2090–2110), explaining the shared target and differences in policy fitting, critic learning, and regularization.
Relation to MORE and REPPO. MORE derives the optimal entropy- and KL-regularized target distribution. Writing its objective in policy notation, this target is
\pi_0(a\mid s)^{\lambda/(\lambda+\tau)} \exp\\!\left(\frac{Q(s,a)}{\lambda+\tau}\right),$$ where $\pi_0$ is the old policy, $\lambda$ controls KL regularization, and $\tau$ the entropy term. For fixed critic and multipliers, the corresponding policy optimization is equivalent to reverse-KL fitting: $$\begin{aligned} \min_\theta\\; (\lambda+\tau)\\, \mathbb E_{s\sim d}D_{\mathrm{KL}}\\!\left(\pi_\theta(\cdot\mid s)\\|p^*(\cdot\mid s)\right) \quad\Longleftrightarrow\quad \max_\theta\\;\mathbb E_{s\sim d}\Big[ &\mathbb E_{a\sim\pi_\theta}[Q(s,a)]\\\\ &+\tau\mathcal H(\pi_\theta(\cdot\mid s)) -\lambda D_{\mathrm{KL}}(\pi_\theta\\|\pi_0) \Big]. \end{aligned}$$ A modern neural-network adaptation of MORE would optimize this reverse-KL objective, whereas FERPO fits the target using forward KL, $D_{\mathrm{KL}}(p^*\\|\pi_\theta)$. See [1], Section 2. REPPO is related to this regularized expected-value optimization, but uses the opposite trust-region direction, $D_{\mathrm{KL}}(\pi_{\mathrm{old}}\\|\pi_\theta)$, and switches between the maximum-entropy objective inside the KL bound and a KL-only penalty outside it (Appendix A.6, lines 2103–2105). This switch brings REPPO’s loss shape closer to FERPO’s in Figure 6. MORE and FERPO still use different KL directions and can produce different updates despite their shared target. **Relation to V-MPO.** V-MPO normalizes weights across the full batch of state–action pairs, whereas FERPO normalizes candidate-action weights separately within each state. This changes how strongly each state contributes to policy fitting. To isolate this difference, hold the conditional target $p^*(a\mid s)$ fixed. Let $d(s)$ be the rollout state distribution, $Z(s)$ the target's state-dependent normalizer, and $$\ell_s(\theta) =-\mathbb E_{a\sim p^*(\cdot\mid s)} [\log\pi_\theta(a\mid s)]$$ the fitting loss at state $s$. Writing $L_{\mathrm{V\text{-}MPO}}$ for the batch-normalized fitting objective in this comparison gives $$\boxed{ \begin{aligned} L_{\mathrm{FERPO}}(\theta) &=\mathbb E_{s\sim d}[\ell_s(\theta)],\\\\ L_{\mathrm{V\text{-}MPO}}(\theta) &=\mathbb E_{s\sim d}\\!\left[ \frac{Z(s)}{\mathbb E_{s'\sim d}[Z(s')]} \ell_s(\theta) \right]. \end{aligned} }$$ FERPO therefore preserves the rollout state weighting, while batch-wide normalization gives greater weight to states with larger $Z(s)$. This difference remains even with unlimited samples. We explain this normalization difference and the additional components of full V-MPO in more detail in Appendix A.6.1 (lines 2112–2165). **Additional comparisons.** We evaluated MORE-inspired and V-MPO variants on four DMC tasks at approximately 50M interactions, alongside FERPO and REPPO’s main-benchmark results. The MORE-inspired adaptation optimizes the entropy-regularized actor objective with an always-active reverse-KL trust-region penalty. For a fair comparison, the MORE and V-MPO variants use matched network sizes, learning rates, rollout settings, and optimization budgets (Appendix A.6.2). The final evaluation returns are IQMs across seeds, including all four completed V-MPO seeds per task: | Task | FERPO | REPPO | MORE-inspired | V-MPO variant | |---|---:|---:|---:|---:| | AcrobotSwingup | **500.9** | 345.5 | 366.0 | 304.9 | | AcrobotSwingupSparse | **36.3** | 7.6 | 5.0 | 0.5 | | FingerSpin | **736.8** | 649.9 | 575.1 | 711.7 | | FishSwim | **724.3** | 643.4 | 127.0 | 137.1 | FERPO achieves the highest reported final return on all four tasks. REPPO outperforms the MORE-inspired variant on three of four tasks, supporting the performance benefits of REPPO’s design choices in this comparison. Appendix A.6.2 presents learning curves (Figure 20) and final returns (Table 7); Section 5.3 summarizes the results. **References** [1] Abdolmaleki et al. “Model-Based Relative Entropy Stochastic Search.” NeurIPS, 2015. [2] Song et al. “V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control.” ICLR, 2020.