- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes multi-step implicit value estimation, which introduces IQL's expectile-based ("optimistic") value estimation into on-policy PPO. The motivation is that sparse rewards lead to a massive volume of zero-value targets and bias the shared critic towards zero. When updating the value function, positive TD errors propagate with a larger weight τ and negative ones with a smaller weight (1−τ). Then the authors prove that the population implicit Bellman operator is a γ-contraction, so the recursive implicit value pair ($Q_π^{\tau}, V_π^{\tau} $) exists and is unique, and exact value iteration converges geometrically. Experiments cover sparse-reward navigation, stochastic MDPs, and dense-reward control, showing gains over PPO+GAE.
Critical strengths
-
The idea of introducing optimistic operation into the GAE of PPO is straightforward. Rare successes should not be washed out by a critic that averages over a mostly failing policy.
-
The algorithm is simple and practical, while the theoretical analysis is detailed.
-
The paper proves that is a γ-contraction (Theorem 3), giving existence, uniqueness, and geometric value iteration for the recursive implicit value pair. It seems to be a solid theoretical contribution.
Critical weaknesses
-
The baselines are insufficient. PPO+GAE is the only baseline. There is no comparison to methods that target the same problem, such as self-imitation learning or Highway RL.
-
The effect of the critic ensemble is not isolated. IV/IQ use a min-ensemble of 2 critics, while PPO uses one. Fig. 6 shows the ensemble is decisive. On Empty-16×16, IV with N = 1 is worse than PPO. GAE with a min-ensemble (τ = 1/2, N = 2) is missing, so the gains cannot be attributed to the expectile.
-
Generally, the writing is hard to follow, as many theorems lack examples and more detailed explanations. In Figure 1, the red arrow overlaps with the text.
Questions for authors
-
Could the authors compare the proposed method with optimistic RL methods?
-
Could the authors report GAE with a min-ensemble of 2 critics (τ = 1/2, N = 2) on the main tasks? Without it, the gains cannot be separated between the expectile and the ensemble.
-
Why does IV with N = 4 collapse on Empty-16×16 (Fig. 6)? How should N be chosen in practice?
6 comments
Thanks the authors for the reply
Thank you for the additional experiments and clarifications. The stronger PPO baselines, Highway RL comparison, and expanded Atari evaluation strengthen the empirical evidence. In particular, the ensemble controls help show that the gains on the six Atari environments cannot be explained by adding critics alone.
However, the response does not directly address the comparison with optimistic RL methods. Matched ensemble controls are also missing on the original sparse-reward tasks, particularly Empty-16×16, where my attribution concern arose. The explanation for the N=4 collapse is plausible but remains unverified on that task.
While I appreciate these improvements, my concerns are only partially resolved. I would like to maintain my weak-reject rating.
Updated
Thanks for engaging.
Optimistic RL
Maybe, the word optimistic is a bit confusing, despite that we said it in several places our optimism is different. I thought HRL is what you asked for as other methods. There are two points that we emphasize again here. We are optimistic w.r.t. policy choice not environment uncertainty. **We improved the exploitation, not exploration. **
DeepSea experiments have shown that our method is not about an improvement on bonus, and is complementary to it. In fact, on 48, IQ alone does not solve the problem, and we include a discussion on Atari games, where our method does not improve the outcome on the montezuma revenge, which again shows our mechanism is different from more well established optimistic exploration RL.
That being said, if you still want a comparison, please name the method you think we need to compare. We expect that there could be complementary gain. The problem is that intrinsic bonus is not that easy to use, because their bonus needs to be combined with the actual reward, which needs scale match.
Empty-16×16 problem
It is a fact that taking minimum over many estimators result in too much perssimism, and in practice that people doing double Q learning use 2 critics, which is what we found to be reliable anyway. Taking N to infinity, you then take minimum, for any random variable without lower bound, you get inifinite small target value. It is a simple mathematical fact. What's wrong with getting worse at 4?
To be honest, we actually do not understand why this is an issue, we are not suggesting bigger N is always better. It is not our central assumption, it is not even our assumption. In fact, we double checked the paper, it is expected in the methodology section to have pessimism when N is too big, even in the first submission. The figures also have captions that marked this observation out. What is there to discuss?
Are you suggesting that we should include an actual value plot? How can we answer your question?
Updated with Running Example in Appendix
Many thanks for the careful read and present a detailed technical summary.
An update on Atari experiments
We conduct a further large scale experiments over 104 Envpool Atari environments, our implicit value estimator achieved higher mean return on 75 games. With statistical significance our method get 55/6 of win/loss against PPO with hyper-parameters from CleanRL See the updated paper for more.
Re W1 "The baselines are insufficient. "
It is well established that PPO has been the de-facto method for multi-step value estimation. Many early methods use sophisticated importance weighting on value, they are even less stable. till, to address the concern about baselines, we conducted additional experiments on six Atari environments to examine whether stronger PPO configurations or a similarly motivated multi-step method could match IQ.
We first strengthen PPO through critic ensembles, alternative aggregation rules, and a larger GAE λ. Indeed, there are some modest gains. We therefore use the best tested PPO configuration separately for each environment in the subsequent comparison. All results below report mean ± SD across eight seeds at 10M steps. PPO variants use GAE λ=0.95 unless otherwise indicated.
Table 1. Stronger PPO configurations.
We additionally evaluate Highway Reinforcement Learning (HRL), a related multi-step method, using minimum and mean critic aggregation. We could not identify an accessible official implementation, so these results use our own implementations. Table 2 compares these variants and the strongest tested PPO baselines with IV and IQ. IQ achieves a higher final mean than the best tested PPO and HRL variants on all six environments.
Table 2. Best tested PPO, our HRL adaptations, and IV/IQ.
See the updated Appendix for more discussion.
Re W2: " the critic ensemble"
To address overestimation caused by function approximator as in double Q learning, we introduced extra critics ensemble in Overestimation and minimum ensembles paragraph. As in double Q learning, too many critics can be harmful. In practice, we find N=2 would be a reliable choice, and we run that option for all 104 Atari games.
The VALUE-ENSEMBLE SIZE section confirms that PPO does not benefit from minimum of critics, as it does not have an overestimation problem. The Table 1 in Re W1 provides additional analysis of using mean of ensemble for PPO did not close the performance gap.
##Re W3: "hard to follow" Thanks for catching the overlap issue, it has now been fixed. We added a more detailed overview at the begining of Section 3 and a theorem map at the begining of Appendix to guide the reader. We further added a section of running example as you requested in Appendix, and adjusted the wording of theorems. We did have remarks explaining the utility of each theorems. Hopefully, with the new map and examples, readers can know what happens intuitively.
We have to maintain that our theoretical results were presented in a thoughtful way, and developed in a logical fashion. Actually, reviewer ixyD finds the paper easy to read.
Re Q1 & Q2
See ReW1 and W2
Re Q3
N=2 would be a safe choice as in double Q learning. It breaks down when N is too big due to over-persmissim. Taking a minimum over a large N random variables will result in a very low value.