acceptodds
Under review as a conference paper at ICLR 2027

Is Exploration or Optimization the Problem for Deep Reinforcement Learning?

Abstract

In the era of deep reinforcement learning, making progress is more complex, as the collected experience must be compressed into a deep model for future exploitation and sampling. Many papers have shown that training a deep learning policy under changing state and action distributions leads to suboptimal performance, or even collapse. This naturally raises the concern that even if the community develops improved exploration algorithms or reward objectives, will those improvements fall on the deaf ears of optimization difficulties? This work proposes a new pracitcal sub-optimality estimator to determine optimization limitations of deep reinforcement learning algorithms. Through experiments acrossenvironments and RL algorithms, it is shown that the difference between the best data generated is better than the policies' learned performance. This large difference indicates that deep RL methods only exploit half of the good experience they generate.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_XhXLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the paper tries to understand when deep RL performance is poor, whether that is caused by lack of good experience due to exploration issues, or lack of ability to exploit good experience. prior work measures optimality gap relative to an oracle policy or human policy, which conflates the two, so this work introduces the notion of experience-best policy. they show a significant gap pointing to poor exploitation on both deterministic and stochastic environments. they also show the gap increases with RND and with scale.

Critical strengths

I think the idea of optimality gap relative to experience is nice and does untangle whether the issue is poor experience or poor exploitation. If measured appropriately it could be a valuable diagnostic for practitioners and the research community. However see weakness section for discussion on rewarding good outcomes vs decisions.

Critical weaknesses

I believe the paper does not adequately address the crucial issue of disentangling luck from better policy in computing the experience optimal policy. Do we reward good outcomes or good decisions? I think we should reward good decisions here but it seems like the proposal is rewarding good outcomes which could be entirely luck in the worst case. They simply define the experience optimal policy return by replaying the top k% best trajectories (possibly recent). One can imagine a simple MDP where taking action A deterministically transitions to terminal state and gives reward 1, while taking action B is a lottery between reward 10 with probability (k+eps)% and reward -inf with remaining probability. Here the experience optimal policy would collect reward 10 by luck. The optimal policy would take action A and collect reward 1.

Also it seems like we need to track 2 numbers to achieve the paper's stated aim, not just the gap between best experience and learned policy, but also the value achieved by the learned policy (or best experience). Just looking at the gap is uninformative, which the authors point out and say they also check to make sure the policy learns, but I think a binary check is insufficient and we need both numbers to compare.

Questions for authors
  • in figure 4 are you tracking the raw gap between experience optimal and learned policy? what about the normalized gap relative to experience optimal? shouldn't the ratio be what matters, not the raw value?
  • how do you address the luck issue?
  • could something like the following help disentangle luck vs good decisions? resample the highest value trajectories (replay those actions where possible until things diverge) and if there's regression to the mean that points to luck?
Reviewer_zJUhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the performance of deep reinforcement learning algorithms is limited by insufficient exploration or by ineffective learning from collected experience. It proposes a “practical sub-optimality” metric comparing average policy performance with high-return trajectories generated during training, considering both historical and recent experience. The experiments examine this gap across several environments and algorithms, including under exploration bonuses and network scaling. The paper raises an important question and provides useful observations about the difference between discovering successful behaviour and reproducing it consistently.


update: raised my score

Critical strengths

(S1: Extensive related work) The extensive related work is a clear strength. It connects exploration, training stability, network scaling, and convergence, placing the research question within a broad and relevant literature.

(S2: Practical metric based on the agent’s own experience) The proposed practical sub-optimality metric provides a simple way to compare average policy performance with successful experience generated during training, without requiring an expert policy or knowledge of the optimal return. Averaging the highest-return episodes reduces reliance on a single exceptional trajectory. The historical and recent variants distinguish past achievements from successful behaviour appearing in current experience, while the normalised version provides a way to summarise results across tasks. This measurement approach offers useful information beyond conventional average-return learning curves.

(S3: Informative empirical comparisons) The main experiments show that the gap varies across tasks and algorithms, and examine its behaviour under exploration bonuses and network scaling. The comparisons between stochastic and deterministic policy evaluation are also valuable, since they help reveal when exploratory action selection contributes to the observed gap.

Critical weaknesses

(W1: Limited validation of the central hypothesis) The paper interprets the gap as evidence of limitations in learning from collected experience and uses it to diagnose optimisation difficulties. The experiments demonstrate that the gap exists and changes under exploration bonuses and network scaling. However, these interventions also change the experience distribution and other aspects of learning, making it difficult to determine whether the gap specifically tracks optimisation difficulty. The deterministic-policy comparisons provide useful controls, but they do not establish how much of the remaining gap can be recovered by learning more effectively from the available data. A controlled intervention that improves average policy performance using the same experience, with the reference benchmark held fixed, would provide more direct support for this interpretation.

(W2: The gap does not establish the dominant bottleneck) Even if some of the gap is recoverable through better learning, this does not establish whether improving learning would be more beneficial than further exploration. The diagnostic claims in Sections 4 and 5.4 therefore require evidence beyond the observed gap itself. In particular, it remains unclear whether the metric predicts the relative benefit of collecting better experience versus learning more effectively from existing experience.

Questions for authors

My assessment is very close to the boundary between weak accept and weak reject, and I'm very open to raising my score substantially if the authors can address the key concerns below. Q is questions and C is comments.

(Q1: Using existing experience) I would like to understand whether changing how the learner uses its collected experience can improve its average return.

I think one experiment that might help is to save DQN checkpoints and their collected experience on MinAtar, then continue training on that experience using standard replay versus more frequent sampling of successful trajectories, for example the highest-return 10%. Keeping other settings the same, does sampling these trajectories more often improve average returns? Do larger initial gaps predict greater improvements? A positive result would provide more direct support for the idea that the agent can improve by learning more effectively from experience it already has.

Repeating this comparison with other equal-sized datasets collected using standard versus enhanced exploration could also help assess whether better experience or better learning provides the larger benefit.

(Q2: Initial states and policy evaluation) How is the top-5% return estimate computed, when initial states vary? Are trajectories from different initial states combined and sorted by return before selecting the top 5%, or is this selection performed separately for each initial state? My concern is that the highest-return trajectories could disproportionately come from initial states where high returns are easier to obtain. Could you explain how initial states are matched between trajectory replay and learned-policy evaluation? It would also help to specify the number of evaluation episodes and the exploration settings used.

(Q3: Interpretation in stochastic environments) To my understanding, the Behavioral Achievability Assumption in Appendix D requires high-return trajectories to outperform the expected return of random actions. I do not see how this condition rules out a gap caused by unavoidable environmental randomness, although the deterministic controls in Appendix C are useful for the corresponding experiments.

For example, consider two actions: A yields a return uniformly distributed on [0,1][0,1], while B always yields zero. Always choosing A is optimal, with expected return 0.5 and top-5% mean 0.975, whereas uniformly random action selection obtains expected return 0.25. The stated assumption therefore holds even though no improvement in expected return is possible. Could you clarify what additional condition would distinguish this case from an optimisation limitation? I would also appreciate a clearer indication of which experiments use fully deterministic environments and which rely on the stochastic interpretation.

(Q4: Sensitivity to the percentage of highest-return episodes) How much the conclusions depend on the percentage of highest-return episodes used to calculate practical sub-optimality? If the estimate uses the top 5%, 10%, or 20% of episodes, do the same algorithms still have smaller practical sub-optimality in Section 5.4? Does adding RND or increasing network size still increase the gap? I would find the metric more convincing if these findings remained consistent across these choices.

(C1: Sign in Equation (3)) Equation (3) appears to have a sign error. Under the standard definition Aπ′=Qπ′−Vπ′A^{\pi'}=Q^{\pi'}-V^{\pi'}, the right-hand side equals Vπ−Vπ′V^\pi-V^{\pi'}, whereas the left-hand side has the opposite sign.

(C2: Notation in Equation (4)) I think for Equation 4, it should be ⟨a0,…,aT⟩\langle a_0,\ldots,a_T\rangle in the argmax subscript.

Reviewer_GknBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the situation when a deep RL agent is performing poorly, is the main problem that it has not discovered sufficiently good behavior, or that it has discovered good behavior but failed to learn to reproduce it? The authors propose a "practical sub-optimality" measure that compares the learned policy's performance with high-return trajectories generated during training. In deterministic environments, they can directly replay the best action sequence the agent has previously generated. In stochastic settings, they use softer versions based on the top (k%) of lifetime or recent returns. They evaluate this across PPO, DQN, PQN, SAC, several Atari/MinAtar and MuJoCo environments, an RND exploration experiment, network scaling, and an RLVR/GRPO example.

Critical strengths

The first strength for me is the empirical phenomenon itself. On several difficult tasks, the agent's trajectory distribution contains behavior that is substantially better than its normal performance. This is useful information that would be invisible if one only plotted average return.

I also appreciated the deterministic-environment controls. Removing environment randomness and replaying previously generated action sequences makes the basic observation much more convincing: in at least some settings, the agent has genuinely executed a high-return sequence and later fails to reproduce comparable performance consistently. The paper also covers a reasonably broad collection of algorithms and environments rather than showing the effect in one isolated setup.

Critical weaknesses

My main concern is that I do not think a large value of the proposed gap is sufficient to diagnose an optimization failure.

For the soft estimator, the appendix makes clear that the quantity is essentially the difference between the upper tail of the return distribution and its mean. A large gap therefore tells us that the agent sometimes performs much better than it usually does. That is interesting, but several mechanisms can produce this without the underlying optimizer failing to learn from available data. For example, imagine that an action is already optimal but produces reward 100 with probability 0.1 and reward 0 otherwise. So a perfectly optimized policy that always chooses this action has expected return 10, while its top 5% return is 100. The proposed gap is therefore very large despite there being no optimization failure. This example also seems to satisfy the "behavioral achievability" condition in Appendix D as long as random actions perform worse. Thus I do not think the current assumption is sufficient to obtain the diagnostic interpretation claimed in the paper.

The deterministic experiments remove one source of this ambiguity, which is valuable, but not all of it. Policy stochasticity or intentional exploration can itself produce a best-versus-average gap. In fact, the paper notes cases where ϵ\epsilon-greedy exploration knocks DQN off an otherwise good path and the deterministic policy performs much better. I would not describe that particular gap as evidence of a network optimization problem.

My second concern is that the experiments mostly show that the gap exists rather than validating what the gap diagnoses. I would have found a fixed-data intervention particularly convincing like collect the experience once, then change only the learning procedure. For example, train a fresh network on the same data, substantially increase the number of updates, reset the representation, or explicitly upweight/imitate the high-return trajectories. If these interventions close the gap without collecting any new experience, that would much more directly support the claim that the bottleneck is exploitation/optimization.

This also affects my view of the the RND result. RND deliberately broadens the behavior/state distribution. Since the proposed statistic depends on the upper tail relative to average behavior, increasing exploration can increase the gap even if the optimizer itself has not become worse. I therefore do not think the larger RND gap alone establishes that adding exploration has aggravated an optimization problem.

Questions for authors
  1. What additional assumptions are needed for a large practical sub-optimality gap to imply that optimization/exploitation is the bottleneck rather than simply indicating a broad or stochastic return distribution?

  2. In the RND experiments, how much of the increased gap can be explained simply by increased return variance or policy stochasticity?

Reviewer_zzDhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a practical sub-optimality estimator to ask whether poor deep-RL performance is caused more by weak exploration or by failure to exploit generated experience. The estimator compares a learned policy with an “experience optimal” reference computed from the best or top-k% observed trajectories; in stochastic settings it corresponds to an upper-tail return statistic. Across PPO, DQN, PQN, SAC, and a GRPO-based RLVR example, the paper reports large gaps between top generated returns and average policy performance, and finds that RND exploration bonuses and larger networks often enlarge this gap.

Critical strengths
  • The diagnostic question is practical and is not directly answered by mean return. Mean return alone cannot distinguish an algorithm that never discovers high-value behavior from one that discovers it but cannot consistently reproduce it. Framing the gap relative to the agent’s own experience gives practitioners a useful complement to human- or oracle-normalized scores.
  • The analysis spans several regimes. The paper studies deterministic replay, stochastic top-k estimation, multiple Atari/MinAtar/MuJoCo tasks, four conventional RL algorithms, exploration bonuses, network scaling, and an RLVR example. The top-k sensitivity analysis and upper-CVaR interpretation are useful steps toward making the metric less anecdotal.
Critical weaknesses
  • The “experience optimal policy” is trajectory replay or an upper tail of observed returns, not a deployable policy; top trajectories may arise from favorable initial states, environment noise, exploratory stochasticity, or policies from earlier in training. Even deterministic replay removes environment randomness without showing that a single callable policy can reproduce the behavior. A large upper-CVaR-to-mean gap therefore demonstrates return variability and promising experiences, but not by itself that a better optimizer could convert those experiences into a consistent policy.
  • The claim that deep RL is more often limited by exploitation than exploration is drawn mainly from chosen difficult settings, four seeds in the main per-task protocol, aggregate rliable plots, and no significance tests for the central claim. RND and larger networks increasing the gap is also consistent with more diverse or risk-seeking exploration rather than worse optimization. Without an intervention such as offline policy extraction, return-conditioned imitation, or a stronger optimizer that reduces the gap, the metric remains descriptive rather than causal.
Questions for authors
  • Can policy extraction, return-conditioned behavior cloning, or offline RL trained on the same generated data substantially close the gap? Such an intervention would directly test exploitability.
  • Can you report confidence intervals over more seeds, use consistent top-k choices, and evaluate a broader pre-specified environment suite rather than selected difficult tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.