acceptodds
Under review as a conference paper at ICLR 2027

Learning from Successes with Multi-Step Implicit Values

Abstract

Discovering a successful trajectory does not ensure that a reinforcement learning agent learns effectively from it. In long-horizon, sparse-reward tasks, evaluating the current policy can assign low values even to states on successful trajectories, weakening the learning signal from rare successes. Building on Implicit Q Learning, we propose multi-step implicit value estimation, which combines expectile-based value learning with recursive temporal propagation. Rather than evaluating the behavior policy alone, the formulation targets an implicit policy that favors higher-valued actions while retaining unbiased expectations over rewards and transitions. We establish contraction of a joint action–state value operator and show that its multi-step compositions share a unique fixed point equal to the expected-return value of this implicit policy under the original MDP. We introduce \IQ, a coupled backward recursion for action and state values, and \IV, a value-only specialization for deterministic environments who recovers generalized advantage estimation (GAE) at the symmetric expectile. Experiments show improvements over PPO with GAE on multiple sparse- and dense-reward tasks, with complementary gains from exploration bonuses. Ablations highlight the importance of multi-step propagation and the versality of our methods.

Primary area:
reinforcement learning
Keywords:
Online reinforcement learning, implicit Q-learning, multi-step value estimation, expectile regression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GJ3iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes multi-step implicit value estimation, which introduces IQL's expectile-based ("optimistic") value estimation into on-policy PPO. The motivation is that sparse rewards lead to a massive volume of zero-value targets and bias the shared critic towards zero. When updating the value function, positive TD errors propagate with a larger weight τ and negative ones with a smaller weight (1−τ). Then the authors prove that the population implicit Bellman operator is a γ-contraction, so the recursive implicit value pair ($Q_π^{\tau}, V_π^{\tau} $) exists and is unique, and exact value iteration converges geometrically. Experiments cover sparse-reward navigation, stochastic MDPs, and dense-reward control, showing gains over PPO+GAE.

Critical strengths
  1. The idea of introducing optimistic operation into the GAE of PPO is straightforward. Rare successes should not be washed out by a critic that averages over a mostly failing policy.

  2. The algorithm is simple and practical, while the theoretical analysis is detailed.

  3. The paper proves that Tπτ=TQ∘TVτT_π^{\tau}=T_Q∘T_V^{\tau} is a γ-contraction (Theorem 3), giving existence, uniqueness, and geometric value iteration for the recursive implicit value pair. It seems to be a solid theoretical contribution.

Critical weaknesses
  1. The baselines are insufficient. PPO+GAE is the only baseline. There is no comparison to methods that target the same problem, such as self-imitation learning or Highway RL.

  2. The effect of the critic ensemble is not isolated. IV/IQ use a min-ensemble of 2 critics, while PPO uses one. Fig. 6 shows the ensemble is decisive. On Empty-16×16, IV with N = 1 is worse than PPO. GAE with a min-ensemble (τ = 1/2, N = 2) is missing, so the gains cannot be attributed to the expectile.

  3. Generally, the writing is hard to follow, as many theorems lack examples and more detailed explanations. In Figure 1, the red arrow overlaps with the text.

Questions for authors
  1. Could the authors compare the proposed method with optimistic RL methods?

  2. Could the authors report GAE with a min-ensemble of 2 critics (τ = 1/2, N = 2) on the main tasks? Without it, the gains cannot be separated between the expectile and the ensemble.

  3. Why does IV with N = 4 collapse on Empty-16×16 (Fig. 6)? How should N be chosen in practice?

6 comments
AuthorsOpenReview ↗

Updated with Running Example in Appendix

Many thanks for the careful read and present a detailed technical summary.

An update on Atari experiments

We conduct a further large scale experiments over 104 Envpool Atari environments, our implicit value estimator achieved higher mean return on 75 games. With statistical significance our method get 55/6 of win/loss against PPO with hyper-parameters from CleanRL See the updated paper for more.

Re W1 "The baselines are insufficient. "

It is well established that PPO has been the de-facto method for multi-step value estimation. Many early methods use sophisticated importance weighting on value, they are even less stable. till, to address the concern about baselines, we conducted additional experiments on six Atari environments to examine whether stronger PPO configurations or a similarly motivated multi-step method could match IQ.

We first strengthen PPO through critic ensembles, alternative aggregation rules, and a larger GAE λ\lambda. Indeed, there are some modest gains. We therefore use the best tested PPO configuration separately for each environment in the subsequent comparison. All results below report mean ± SD across eight seeds at 10M steps. PPO variants use GAE λ=0.95\lambda=0.95 unless otherwise indicated.

Table 1. Stronger PPO configurations.

EnvironmentPPO 1 criticPPO 2 critics minPPO 2 critics meanPPO 4 critics meanPPO 2 critics mean λ=.999\lambda=.999
Asteroids2189.22 ± 1323.821325.00 ± 245.272184.38 ± 493.412490.94 ± 751.401532.19 ± 204.97
BeamRider2463.56 ± 1110.03726.88 ± 149.882755.41 ± 942.452863.16 ± 802.421104.38 ± 224.32
DoubleDunk-8.69 ± 5.16-6.03 ± 5.53-1.69 ± 1.40-1.72 ± 1.170.75 ± 2.28
Frostbite260.16 ± 9.02265.94 ± 7.31537.66 ± 496.291086.09 ± 1444.041051.41 ± 1165.07
Jamesbond178.91 ± 95.84165.62 ± 110.19529.69 ± 44.79591.41 ± 295.86482.03 ± 20.71
Zaxxon9173.44 ± 1291.7267.19 ± 130.755260.94 ± 3420.806998.44 ± 3264.788381.25 ± 714.11

We additionally evaluate Highway Reinforcement Learning (HRL), a related multi-step method, using minimum and mean critic aggregation. We could not identify an accessible official implementation, so these results use our own implementations. Table 2 compares these variants and the strongest tested PPO baselines with IV and IQ. IQ achieves a higher final mean than the best tested PPO and HRL variants on all six environments.

Table 2. Best tested PPO, our HRL adaptations, and IV/IQ.

EnvironmentBest PPOHRL minHRL meanIVIQ
Asteroids2490.94 ± 751.401154.69 ± 129.701254.06 ± 215.417828.44 ± 12727.624663.44 ± 3990.39
BeamRider2863.16 ± 802.42951.00 ± 310.452230.78 ± 998.854335.97 ± 1028.435797.38 ± 972.15
DoubleDunk0.75 ± 2.28-1.75 ± 1.16-15.16 ± 2.66-0.25 ± 1.807.88 ± 9.85
Frostbite1086.09 ± 1444.04278.91 ± 13.27317.81 ± 136.34801.25 ± 705.152424.84 ± 1980.74
Jamesbond591.41 ± 295.86462.50 ± 68.95382.81 ± 54.87716.41 ± 262.771589.84 ± 420.64
Zaxxon9173.44 ± 1291.723575.00 ± 2352.12281.25 ± 426.994268.75 ± 3539.069912.50 ± 6503.91

See the updated Appendix for more discussion.

Re W2: " the critic ensemble"

To address overestimation caused by function approximator as in double Q learning, we introduced extra critics ensemble in Overestimation and minimum ensembles paragraph. As in double Q learning, too many critics can be harmful. In practice, we find N=2 would be a reliable choice, and we run that option for all 104 Atari games.

The VALUE-ENSEMBLE SIZE section confirms that PPO does not benefit from minimum of critics, as it does not have an overestimation problem. The Table 1 in Re W1 provides additional analysis of using mean of ensemble for PPO did not close the performance gap.

##Re W3: "hard to follow" Thanks for catching the overlap issue, it has now been fixed. We added a more detailed overview at the begining of Section 3 and a theorem map at the begining of Appendix to guide the reader. We further added a section of running example as you requested in Appendix, and adjusted the wording of theorems. We did have remarks explaining the utility of each theorems. Hopefully, with the new map and examples, readers can know what happens intuitively.

We have to maintain that our theoretical results were presented in a thoughtful way, and developed in a logical fashion. Actually, reviewer ixyD finds the paper easy to read.

Re Q1 & Q2

See ReW1 and W2

Re Q3

N=2 would be a safe choice as in double Q learning. It breaks down when N is too big due to over-persmissim. Taking a minimum over a large N random variables will result in a very low value.

Reviewer_GJ3iOpenReview ↗

Thanks the authors for the reply

Thank you for the additional experiments and clarifications. The stronger PPO baselines, Highway RL comparison, and expanded Atari evaluation strengthen the empirical evidence. In particular, the ensemble controls help show that the gains on the six Atari environments cannot be explained by adding critics alone.

However, the response does not directly address the comparison with optimistic RL methods. Matched ensemble controls are also missing on the original sparse-reward tasks, particularly Empty-16×16, where my attribution concern arose. The explanation for the N=4 collapse is plausible but remains unverified on that task.

While I appreciate these improvements, my concerns are only partially resolved. I would like to maintain my weak-reject rating.

Thanks for engaging.

Optimistic RL

Maybe, the word optimistic is a bit confusing, despite that we said it in several places our optimism is different. I thought HRL is what you asked for as other methods. There are two points that we emphasize again here. We are optimistic w.r.t. policy choice not environment uncertainty. **We improved the exploitation, not exploration. **

DeepSea experiments have shown that our method is not about an improvement on bonus, and is complementary to it. In fact, on 48, IQ alone does not solve the problem, and we include a discussion on Atari games, where our method does not improve the outcome on the montezuma revenge, which again shows our mechanism is different from more well established optimistic exploration RL.

That being said, if you still want a comparison, please name the method you think we need to compare. We expect that there could be complementary gain. The problem is that intrinsic bonus is not that easy to use, because their bonus needs to be combined with the actual reward, which needs scale match.

Empty-16×16 problem

It is a fact that taking minimum over many estimators result in too much perssimism, and in practice that people doing double Q learning use 2 critics, which is what we found to be reliable anyway. Taking N to infinity, you then take minimum, for any random variable without lower bound, you get inifinite small target value. It is a simple mathematical fact. What's wrong with getting worse at 4?

To be honest, we actually do not understand why this is an issue, we are not suggesting bigger N is always better. It is not our central assumption, it is not even our assumption. In fact, we double checked the paper, it is expected in the methodology section to have pessimism when N is too big, even in the first submission. The figures also have captions that marked this observation out. What is there to discuss?

Are you suggesting that we should include an actual value plot? How can we answer your question?

Reviewer_KxM5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes multi-step implicit value estimation (specifically IV and IQ estimators), which prevents assigning low values even on successful trajectories in sparse reward tasks. This is implemented by having asymmetric weights depending on the GAE delta, having higher weight for positive advantage, lower weight for negative advantage, similar to IQL (with high expectile). Authors provide theoretical motivation for this approach, and experimental results shows that it is able to improve PPO across sparse and dense reward environments.

Critical strengths

S1. Method is simple and intuitive.

  • The method gets optimistic estimates by substituting the MSE learning loss to expectile loss.
  • The proposed method can be seamlessly applied to existing on-policy algorithms, and extending GAE.
Critical weaknesses

W1. Insufficient comparison with well-tuned PPO baselines.

  • The main motivation of this paper is that conventional value estimation struggles to propagate reward signals over long horizons in sparse reward setting.
  • However, it is unclear whether this problem can be addressed by appropriately tuning PPO's discount factor (\gamma), which is designed to deal with the long-horizon value bootstrapping.
  • I encourage the authors to compare against PPO with a hyperparameter sweep over γ\gamma, particularly in the sparse reward environments (e.g., DeepSea-48, Ant Big Maze)

W2. Experimental setting (Section 5.1) and motivation is not aligned.

  • The RND setting introduces dense rewards, not a sparse reward setting that motivates the proposed method.
  • Although combining the proposed method with RND is an interesting experiment, it is somewhat different from the original motivation of learning from sparse successful experiences. In particular, it is unclear whether the improvements come from better propagation of rare extrinsic rewards (which paper mainly claims) or from learning with the dense intrinsic rewards.
  • I do not consider the having these experiments a weakness by itself, but their relevance to the paper's main contribution should be better justified.
Questions for authors

Q1. Comparison with well-tuned PPO baselines.

  • Can the authors provide additional comparisons with PPO using larger discount factors (\gamma)? (If possible, various GAE lambda too)
  • I think this comparison is important for showing the advantage of the proposed method over conventional multi-step value estimation (GAE)

Q2. Clarification on Section 5.1

  • Can the authors clarify what the authors want to show on this experiment?
  • I think it is an interesting experiment, but I couldn’t find a relation between the claim of the paper and this experiment.
6 comments

Thanks for acknowledging the method is simple, intuitive and easy to use.

An update on Atari experiments

We conduct a further large scale experiments over 104 Envpool Atari environments, our implicit value estimator achieved higher mean return on 75 games. With statistical significance our method get 55/6 of win/loss against PPO with hyper-parameters from CleanRL See the updated paper for more.

Updated a theorem map and running example in Appendix to improve readbility

Re Strength:

Re S1: "substituting the MSE learning loss to expectile loss"

This is partly what we did. However, we develop a finer understanding of the implicit values, and the resulting multi-step value propagation is not that trivial. That being said, in the end, the resulting method is simple, intuitive and can be used as a plug-in as you correctly pointed out.

Re Critical Weaknesses:

W1. Insufficient comparison with well-tuned PPO

We swept GAE λ∈{0.95, 0.98, 0.99, 0.999, 1} using 32 matched seeds on DeepSea-48 and Ant Big Maze. IV achieved a higher mean Ant return when GAE λ is controlled :

MethodGAE λDeepSea-48Ant Big Maze
PPO0.950 ± 00.0938 ± 0.5303
PPO0.980 ± 00.3359 ± 1.7685
PPO0.990 ± 00.2344 ± 1.2377
PPO0.9990 ± 01.1406 ± 6.4523
PPO10 ± 00 ± 0
IV0.90 ± 01.2266 ± 4.1796
IV0.9990 ± 02.1641 ± 5.8663

We conducted additional experiments on six Atari environments to examine whether stronger PPO configurations or a similarly motivated multi-step method could match IQ. In Table 1, we strengthen PPO through critic ensembles, alternative aggregation rules, and a larger GAE λ\lambda. All results below report mean ± SD across eight seeds at 10M steps. PPO variants use GAE λ=0.95\lambda=0.95 unless otherwise indicated.

Table 1. Stronger PPO configurations.

EnvironmentPPO 1 criticPPO 2 critics minPPO 2 critics meanPPO 4 critics meanPPO 2 critics mean λ=.999\lambda=.999
Asteroids2189.22 ± 1323.821325.00 ± 245.272184.38 ± 493.412490.94 ± 751.401532.19 ± 204.97
BeamRider2463.56 ± 1110.03726.88 ± 149.882755.41 ± 942.452863.16 ± 802.421104.38 ± 224.32
DoubleDunk-8.69 ± 5.16-6.03 ± 5.53-1.69 ± 1.40-1.72 ± 1.170.75 ± 2.28
Frostbite260.16 ± 9.02265.94 ± 7.31537.66 ± 496.291086.09 ± 1444.041051.41 ± 1165.07
Jamesbond178.91 ± 95.84165.62 ± 110.19529.69 ± 44.79591.41 ± 295.86482.03 ± 20.71
Zaxxon9173.44 ± 1291.7267.19 ± 130.755260.94 ± 3420.806998.44 ± 3264.788381.25 ± 714.11

We further evaluate Highway Reinforcement Learning (HRL), a related multi-step method, using minimum and mean critic aggregation. We could not identify an official implementation, so HRL is our own implementations. Table 2 shows IQ achieves a higher final mean than the best tested PPO and HRL variants on all six environments.

Table 2. Best PPO, our HRL adaptations, and IV/IQ.

EnvironmentBest PPOHRL minHRL meanIVIQ
Asteroids2490.94 ± 751.401154.69 ± 129.701254.06 ± 215.417828.44 ± 12727.624663.44 ± 3990.39
BeamRider2863.16 ± 802.42951.00 ± 310.452230.78 ± 998.854335.97 ± 1028.435797.38 ± 972.15
DoubleDunk0.75 ± 2.28-1.75 ± 1.16-15.16 ± 2.66-0.25 ± 1.807.88 ± 9.85
Frostbite1086.09 ± 1444.04278.91 ± 13.27317.81 ± 136.34801.25 ± 705.152424.84 ± 1980.74
Jamesbond591.41 ± 295.86462.50 ± 68.95382.81 ± 54.87716.41 ± 262.771589.84 ± 420.64
Zaxxon9173.44 ± 1291.723575.00 ± 2352.12281.25 ± 426.994268.75 ± 3539.069912.50 ± 6503.91

See the updated Appendix for more discussion.

W2. Experimental setting (Section 5.1) and motivation is not aligned.

"The RND setting introduces dense rewards, not a sparse reward setting that motivates the proposed method."

The setting is actually central to our study. The reviewer might not be very familiar with the reward bonus/intrinsic bonus approach as a solution to sparse reward problem. RND and methods a like is introducing dense signal so that the agent can still learn something if no final reward has every being seen. Those intrinsic bonus will guide the agent to explore novel regions so that they can work for sparse reward setting.

We add RND so that to make it clear that our contribution is orthogonal to the exploration problem. Without RND, neither IQ nor PPO can work in Deepsea-48, yet with RND, IQ can improve over PPO, this experiment shows that our implicit value improve the algorithm via better exploitation of successful outcomes, and it is complementary to the improved exploration. We also slightly updated the description and titles in the paper.

#Re Q: See Re Weakness

Thank you for the clarification and additional experiments. Those clarified the concerns that I had.

To double-check, is W1 result with γ\gamma sweep, or λ\lambda sweep?

Thanks for the follow up. We are glad that the further update helps.

The sweep is on λGAE\lambda_\mathrm{GAE} , so we are trying to see whether PPO can do better as you suggested. Then, for IV as we fix τ=0.6\tau=0.6, and we have λv=λGAE/τ\lambda_v=\lambda_\mathrm{GAE}/\tau as described in line 300. So, λv\lambda_v reactively changes according to λGAE\lambda_\mathrm{GAE}.

Reviewer_ixyDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores multi-step implicit value estimation for RL, introducing IQ and IV to combine expectile-based value learning with recursive propagation, enabling more effective learning from high-return experiences in long-horizon tasks.

Critical strengths
  1. The idea of combining expectile-based value learning with multi-step propagation is useful, as it provides a plausible way to make better use of successful trajectories, especially in long-horizon tasks.

  2. The theoretical analysis further helps clarify what the proposed estimators learn and how they relate to GAE, while the experiments across different reward settings suggest that the approach can be useful beyond a single type of environment.

Critical weaknesses
  1. The direct comparison between one-step and multi-step variants is primarily conducted on DeepSea. Additional ablation comparisons on representative dense- and sparse-reward tasks would help clarify how much of the observed improvement can be attributed specifically to multi-step propagation.

  2. Although the proposed methods are evaluated across a broad range of environments, the main comparisons primarily focus on PPO. Including comparisons with more closely related multi-step value estimation methods would help better establish the empirical advantages of the proposed approach over existing alternatives.

Questions for authors

Can the authors clarify under what conditions the theoretical guarantees extend to the practical training procedure, and whether the approximate actor updates preserve the policy improvement properties established by the theory?

3 comments

We thank the reviewer for putting out a concise and accurate summary.

An update on Atari experiments

We conduct a further large scale experiments over 104 Envpool Atari environments, our implicit value estimator achieved higher mean return on 75 games. With statistical significance our method get 55/6 of win/loss against PPO with hyper-parameters from CleanRL See the updated paper for more.

Critical Weaknesses:

W1: The direct comparison between one-step and multi-step variants is... Re W1: The one-step comparison is primarily work as a sanity check. It is not quite reasonable to expect the one-step version to work over long horizon. That being said, we performed an additional experiment on Ant Big Maze (an env mentioned by reviewer KxM5).

MethodGAE λReturn
PPO multi-step0.950.0938 ± 0.5303
PPO multi-step0.980.3359 ± 1.7685
PPO multi-step0.990.2344 ± 1.2377
PPO multi-step0.9991.1406 ± 6.4523
IV multi-step0.9992.1641 ± 5.8663
IV one-step0.9990.1563 ± 0.6920

We can see that it is critical that IV use multi-step. It is conceivble that if the environment has short maximum length, one-step algorithm can also work.

W2: '' more closely related multi-step value estimation methods"

ReW2: It is well established that PPO has been the de-facto method for multi-step value estimation. Many early methods use sophisticated importance weighting on value, they are even less stable. Still, to address the concern about baselines, we conducted additional experiments on six Atari environments to examine whether stronger PPO configurations or a similarly motivated multi-step method could match IQ.

We first strengthen PPO through critic ensembles, alternative aggregation rules, and a larger GAE λ\lambda. Indeed, there are some modest gains. We therefore use the best tested PPO configuration separately for each environment in the subsequent comparison. All results below report mean ± SD across eight seeds at 10M steps. Bold marks the highest mean in each row. PPO variants use GAE λ=0.95\lambda=0.95 unless otherwise indicated.

Table 1. Stronger PPO configurations.

EnvironmentPPO 1 criticPPO 2 critics minimumPPO 2 critics meanPPO 4 critics meanPPO 2 critics mean λ=.999\lambda=.999
Asteroids2189.22 ± 1323.821325.00 ± 245.272184.38 ± 493.412490.94 ± 751.401532.19 ± 204.97
BeamRider2463.56 ± 1110.03726.88 ± 149.882755.41 ± 942.452863.16 ± 802.421104.38 ± 224.32
DoubleDunk-8.69 ± 5.16-6.03 ± 5.53-1.69 ± 1.40-1.72 ± 1.170.75 ± 2.28
Frostbite260.16 ± 9.02265.94 ± 7.31537.66 ± 496.291086.09 ± 1444.041051.41 ± 1165.07
Jamesbond178.91 ± 95.84165.62 ± 110.19529.69 ± 44.79591.41 ± 295.86482.03 ± 20.71
Zaxxon9173.44 ± 1291.7267.19 ± 130.755260.94 ± 3420.806998.44 ± 3264.788381.25 ± 714.11

We additionally evaluate Highway Reinforcement Learning (HRL), a related multi-step method, using minimum and mean critic aggregation. We could not identify an accessible official implementation, so these results use our own implementations. Table 2 compares these variants and the strongest tested PPO baselines with IV and IQ. IQ achieves a higher final mean than the best tested PPO and HRL variants on all six environments.

Table 2. Best tested PPO, our HRL adaptations, and IV/IQ.

EnvironmentBest PPOHRL minimumHRL meanIVIQ
Asteroids2490.94 ± 751.401154.69 ± 129.701254.06 ± 215.417828.44 ± 12727.624663.44 ± 3990.39
BeamRider2863.16 ± 802.42951.00 ± 310.452230.78 ± 998.854335.97 ± 1028.435797.38 ± 972.15
DoubleDunk0.75 ± 2.28-1.75 ± 1.16-15.16 ± 2.66-0.25 ± 1.807.88 ± 9.85
Frostbite1086.09 ± 1444.04278.91 ± 13.27317.81 ± 136.34801.25 ± 705.152424.84 ± 1980.74
Jamesbond591.41 ± 295.86462.50 ± 68.95382.81 ± 54.87716.41 ± 262.771589.84 ± 420.64
Zaxxon9173.44 ± 1291.723575.00 ± 2352.12281.25 ± 426.994268.75 ± 3539.069912.50 ± 6503.91

See the updated Appendix for more discussion.

Re Question

If the question is about whether the theory cover the deep reinforcement learning cases being examimed in the experiments, the answer is no. We have cleared marked this in remark 2. Our theory is about population level operator. Our theory does apply to tabular MDP with known distribution and rewards. The sort of setting that introduction to reinforcement learning will start with. We also like to point out that to the best of our knowledge no convergence proof has been given to any practical deep reinforcement learning algorithm.

Reviewer_ixyDOpenReview ↗

Response to authors' rebuttal

Problem solved, and score raised.

Thanks for the quick response and constructive feedbacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.