acceptodds
Under review as a conference paper at ICLR 2027

Balancing Learnability and Bellman-Error Coverage in Prioritized Experience Replay

Abstract

Experience replay improves the sample efficiency of off-policy reinforcement learning, but its benefit depends on which transitions are revisited. Prioritized experience replay (PER) favours large temporal difference (TD) errors, which may overemphasize noisy or irreducible samples. Reducible-loss prioritization (ReLo) instead targets errors that appear learnable, yet it can fail in the complementary direction: when the reference loss is imperfect, difficult but informative transitions may receive negligible priority and disappear from replay. We introduce BLCR (balanced learnability–coverage replay), a conservative repair that combines a normalized TD-error priority floor with bounded reducible-loss weighting. Unlike an additive mixture, the floor is activated only when ReLo underprioritizes a high-error transition; otherwise, the reducible-loss ranking is preserved. Across five MinAtar games at two million frames and five seeds, BLCR improves over ReLo on every game and obtains the highest final return on Asterix and Breakout. Across 30 ALE-RAM games, it outperforms ReLo on 25 games and wins 118 of 150 paired-seed comparisons. In a preliminary 300K-step SAC extension over four MuJoCo locomotion tasks, BLCR-SAC ranks first on Hopper and Walker2d, improving the final return over the SAC by 12.8% and 56.5%, respectively, while the results on Ant and HalfCheetah remain mixed. These findings position conservative TD error coverage as a targeted correction to reduce loss replay rather than a universal replacement for PER.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Replay buffer prioritization, Reducible-loss prioritization, Sample-Efficient reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Bv3wOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Balanced Learnability–Coverage Replay (BLCR), an experience replay method that aims to prevent reducible-loss prioritization (ReLo) from suppressing transitions with large Bellman errors. BLCR combines normalized reducible-loss priorities with a logarithmically compressed TD-error floor and applies bounded per-sample loss weighting. The method is evaluated on MinAtar, 30 ALE-RAM games, and a preliminary SAC extension to MuJoCo. The results show improvements over ReLo across most evaluated discrete-control tasks, while comparisons with PER and uniform replay, where available, yield mixed results.

Critical strengths

None

Critical weaknesses
  1. The presentation is overly repetitive, and the material is not appropriately organized across sections.

  2. The proposed method combines several components, but the rationale for individual design choices is insufficiently justified, and the ablation studies provide only a limited assessment of their respective contributions to performance.

  3. The main text includes excessive detail on minor hyperparameters and implementation settings, which substantially reduces readability. These details would be better presented in the appendix.

  4. The experimental comparisons lack important baselines, particularly uniform replay and PER in the 30-game ALE-RAM benchmark.

  5. Although the proposed method improves performance on some tasks, the results do not establish a consistent advantage over the evaluated baselines.

Questions for authors

See weaknesses.

Reviewer_R1asOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BLCR, a modified prioritized experience replay method that addresses a limitation of Reducible Loss (ReLo) prioritization. ReLo assigns priority according to the estimated learnable part of a sample’s loss. However, its reference loss may be inaccurate, causing samples with large Temporal Difference (TD) errors to receive very low priority. BLCR addresses this problem by imposing a compressed TD-error lower bound on the ReLo priority. The lower bound increases a sample’s priority only when the ReLo signal is too small. BLCR also uses bounded weights to adjust the contribution of sampled transitions to the Bellman loss. The paper evaluates BLCR on five MinAtar games, 30 ALE-RAM games, and four MuJoCo tasks using SAC. The results show that BLCR often improves over ReLo, although it does not consistently outperform PER or other baselines.

Critical strengths
  1. The method provides a clear correction for a specific failure mode of ReLo. Equations 10 and 11 formulate BLCR as a projection of the ReLo priority onto a constraint defined by a TD-error lower bound. Unlike a simple weighted sum of two priority signals, the lower bound is active only when the ReLo priority is too small. This gives a clear connection between the motivation and the algorithm design.
  2. The paper is careful about the scope of its claims. It does not claim that BLCR always outperforms PER or ReLo. The authors report negative or mixed results, including Bank Heist, Kung Fu Master, Fishing Derby, CartPole, LunarLander, and HalfCheetah. These results help clarify when the method is useful and support the paper’s claim that BLCR is a targeted correction rather than a universal replacement for existing replay methods.
  3. The evaluation covers five MinAtar games with five random seeds and 30 ALE-RAM games. The results show that BLCR has higher average returns than ReLo on 25 of the 30 ALE-RAM games and wins 118 out of 150 paired-seed comparisons. This provides useful evidence that the improvement over ReLo is not limited to a small number of selected tasks. However, the ALE-RAM experiments should be interpreted as comparisons with ReLo, since they do not include DQN or PER.
  4. The method has clear mathematical properties. The paper explains when BLCR preserves the ReLo priority, gives a lower bound on the replay probability, and bounds the additional loss weight. These properties make it easier to understand how BLCR changes the replay distribution and the Bellman update.
Critical weaknesses
  1. The paper provides some evidence for the proposed mechanism, including the ReLo-F ablation and the average priority lift in Figure 4(b). However, it does not report the lower-bound activation rate, the replay mass assigned to priority-boosted transitions, or the learning progress of those transitions. Thus, the results show that BLCR can improve over ReLo, but do not establish that the gain comes specifically from fixing TD-error undercoverage.
  2. The 30-game ALE-RAM evaluation compares only ReLo and BLCR. Without standard baselines such as DQN or PER, it shows an improvement over ReLo but does not establish whether BLCR is competitive with commonly used replay methods. The continuous-control evaluation is limited to four MuJoCo tasks, 300K environment steps, and three random seeds. Also, ReLo-SAC and BLCR-SAC use a realized one-step critic-loss reduction proxy, whereas the original ReLo SAC uses a target-critic proxy.
  3. The comparison is internally consistent, but it is not a direct reproduction of the original ReLo SAC setting. These results provide only limited evidence for generalization to continuous action spaces.
  4. The robustness to λ\lambda and η\eta is not sufficiently studied. The paper uses fixed values and does not show how these parameters affect different tasks or training stages. Since the paper suggests that some negative results may depend on the lower-bound strength, a sensitivity analysis would help assess the stability and tuning cost of BLCR.
Questions for authors
  1. Could you provide transition-level diagnostics for the lower-bound mechanism? In particular, please report the lower-bound activation rate, the fraction of replay mass assigned to priority-boosted transitions, and the subsequent change in TD loss or Bellman error for these transitions. These results would help test whether the performance gains come from recovering transitions that ReLo underprioritizes.
  2. For the ALE-RAM experiments, could you include at least one standard replay baseline, such as DQN or PER? If this is not possible, could you state more clearly that the 30-game results demonstrate an improvement over ReLo, rather than an advantage over standard replay methods more generally?
  3. Could you provide a sensitivity analysis for λ and η? It would be helpful to show how these parameters affect lower-bound activation, the priority distribution, and final returns across different tasks and training stages.
Reviewer_m5gLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BCLR, a prioritized experience replay method designed to address a potential failure mode of ReLo. While ReLo prioritizes transitions according to estimated reducible loss, its reliance on an imperfect reference network may assign low priority to transitions that still exhibit substantial TD errors. BLCR addresses this issue by introducing a log-compressed, normalized TD-error floor, combined with bounded loss weighting. The priority is defined through a maximum operator, preserving the ReLo score unless the TD-error floor exceeds it. The paper provides a simple constrained-projection interpretation of this mechanism and evaluates BLCR on five MinAtar games, 30 ALE-RAM games, and four MuJoCo continuous-control tasks. The results show consistent improvements over ReLo on MinAtar and most ALE-RAM games, with mixed but promising results in continuous control.

Critical strengths
  1. A meaningful and well-motivated limitation of reducible-loss prioritization. The paper identifies a plausible failure mode of ReLo: a low estimated reducible loss does not necessarily imply that a transition has little remaining learning value.

  2. A simple and interpretable correction mechanism. The proposed maximum-based priority rule provides a clear, asymmetric intervention: TD error affects a transition's priority only when the reducible-loss estimate falls below the specified floor. This is more targeted than a straightforward additive combination of TD error and reducible loss. The method is relatively simple to implement, and its priority rule and bounded loss weighting are explicitly formulated.

Critical weaknesses

My primary concern is the explanatory gap between the proposed mechanism and the observed performance improvements. The paper argues that BLCR improves performance by restoring replay coverage for transitions that ReLo underprioritizes despite their remaining Bellman errors. However, the empirical analysis primarily demonstrates two observations: BLCR increases some priorities, and BLCR achieves higher returns in several environments. It does not establish the causal connection between them.

Besides, the TD-error coverage guarantee has limited implications in a dynamic replay buffer. Section 3.3 derives a lower bound on the sampling probability from the TD-error floor. While the mathematical derivation appears straightforward, its practical significance is limited. Equation (12) provides an instantaneous probability bound based on the assigned priorities and current buffer size, rather than a guarantee that transitions with large current TD errors will receive sufficient replay throughout training. In particular, Algorithm 1 updates priorities only for sampled transitions. As the online network, target network, normalization statistics, and replay-buffer contents evolve, previously assigned priorities can become stale. A transition that becomes informative after its last priority update may therefore remain underprioritized until it is sampled again.

For experiments, the 30-game ALE-RAM evaluation compares only BLCR against ReLo, omitting both uniform DQN and standard PER. As the motivation for BLCR is partly that ReLo can suppress transitions with large TD errors, improving upon a potentially weak ReLo baseline does not establish that BLCR offers an advantage over standard replay strategies.

Questions for authors

Please see the weakness part above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.