acceptodds
Under review as a conference paper at ICLR 2027

From Discovery to Mastery: Reallocating the Replay Budget in Massively Parallel Reinforcement Learning

Abstract

Massively parallel interaction makes rare successful behavior available for learning, yet discovering these behaviors does not ensure that a policy learns to reproduce them. We identify a discovery–mastery gap: policies learn to reliably reproduce discovered high-quality behaviors only after substantial delays or fail to do so within the training budget. We find that increasing the replay share of successful experience can accelerate learning and enable task success without increasing interaction or update budgets, suggesting that insufficient exposure to such experience contributes to this gap. Motivated by these findings, we propose return-guided reallocation of the replay budget (REAP), a generic sampling mechanism that mixes a base sampling distribution with one proportional to positive centered return-to-go. Its return-based scores are computed once from observed rewards when each episode ends and remain fixed as training proceeds. Across 48 control tasks, REAP enables learning on challenging sparse-reward tasks where baselines struggle and substantially improves sample efficiency on the remaining tasks.

Primary area:
reinforcement learning
Keywords:
massively parallel reinforcement learning, experience replay, non-uniform replay, deep reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cKyyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies replay allocation in massively parallel off-policy reinforcement learning and identifies a “discovery–mastery gap,” where successful trajectories may already exist in replay while the policy still fails to reproduce them reliably. The authors propose REAP, which increases replay probability for transitions with positive centered return-to-go while retaining a base sampler. Experiments across 48 control tasks show improved learning efficiency over uniform replay, TGS, and PER. Presentation

Critical strengths

1.The paper isolates an interesting mismatch between discovering successful behavior and learning to reproduce it, and uses replay-switch interventions to test the role of replay allocation. 2. REAP is simple to integrate with an off-policy learner. Its scores are computed from observed rewards when episodes end, without repeatedly updating priorities from critic estimates. 3. The evaluation is broad, covering 48 tasks across four benchmarks. The paper also reports results with TD3, wall-clock time, and replay-budget ablations.

Critical weaknesses
  1. Unclear distinction between discovery and mastery. The paper uses stochastic evaluation to measure behavior discovery and deterministic evaluation to measure policy mastery. However, the different action-selection strategies may contribute to the observed performance gap, making it difficult to determine whether this gap truly reflects a lack of policy mastery.

2.Effect of episode length on replay priorities. The centered return-to-go depends on the remaining episode length, but the paper does not analyze how this affects replay priorities across episodes of different lengths.

  1. Insufficient implementation details. The paper does not clearly explain how warmup transitions are handled after the reference reward is established. This makes the replay sampling procedure difficult to fully understand and reproduce.

4.Limited comparison with existing replay strategies. Although REAP is compared with Uniform, TGS, and PER, the experiments do not establish its advantages over other episodic or return-based replay mechanisms explored in prior work.

Questions for authors

1.In Figure 3, the authors use deterministic evaluation to measure policy mastery and stochastic evaluation to indicate successful behavior discovery. Could the authors clarify how deterministic actions are selected from the learned policy (e.g., using the mean action)? Furthermore, why is deterministic success rate considered an appropriate measure of mastery, rather than evaluating the reliability of the stochastic policy itself? 2.In Section 3.1, the centered return-to-go depends on the remaining episode length (L_i). How does this affect the sampling probabilities of transitions from episodes of different lengths? Have the authors considered this effect in their experiments? 3.How are warmup transitions handled in REAP, given that rrefr_{\mathrm{ref}} is determined only after warmup? Are they scored retrospectively or sampled only through the base distribution? 4.Several prior studies have explored improved experience replay strategies for continuous-control RL, such as AE-DDPG [1] and ISAC [2]. Could the authors provide further comparisons with these or other relevant replay methods to better evaluate the effectiveness of REAP?

[1]Asynchronous episodic deep deterministic policy gradient: Toward continuous control in computationally complex environments. [2]Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cKyy (1/2)

We thank the reviewer for the careful review and constructive comments. We address the four points below.


W1 / Q1: Discovery and mastery

In Figure 3, successful trajectories occur under stochastic execution, but success remains extremely rare. Across the evaluated checkpoints after initialization, the highest mean stochastic success rates are approximately 0.059% for PlaceSphere within the first 10M steps, 0.234% for PullCube within 5M steps, and 0.012% for LiftPegUpright within 20M steps. Deterministic success rates are zero or near zero over the same respective windows.

We use discovery to indicate that successful trajectories have been generated, and mastery to indicate reliable task completion. A stochastic policy can also demonstrate mastery if it succeeds consistently. Here, the low absolute success rates under both evaluation modes support the conclusion that successful behavior has been discovered but remains unreliable. The difference between action-selection strategies alone would not establish this conclusion.

For deterministic evaluation, we select a=tanh⁡(μθ(s))a=\tanh(\mu_\theta(s)), where μθ(s)\mu_\theta(s) is the Gaussian mean before the tanh transformation. This follows a common SAC practice in which actions are sampled during experience collection and deterministic actions are used for performance evaluation. The resulting success rate measures whether the policy reliably completes the task without action sampling and serves as our operational measure of mastery under this execution protocol.


W2 / Q2: Episode length and replay priorities

Remaining episode length influences sampling through the accumulated centered return. Let Zi=∑k=0Li−1γkZ_i=\sum_{k=0}^{L_i-1}\gamma^k and let rˉiγ\bar r_i^\gamma denote the discounted average reward over the remaining trajectory. The score is si=Zi[rˉiγ−rref]+s_i=Z_i[\bar r_i^\gamma-r_{\mathrm{ref}}]_+, and the sampling probability is pi=λsi/∑jsj+(1−λ)pibasep_i=\lambda s_i/\sum_j s_j+(1-\lambda)p_i^{\mathrm{base}}. At the same positive average excess reward, a larger ZiZ_i therefore gives a transition greater relative weight in the return branch.

We retain cumulative scoring to preserve the total discounted excess reward along each observed continuation, including differences between brief and sustained favorable outcomes. Dividing by ZiZ_i changes the criterion to discounted average excess reward. Discounting bounds the effective-length factor by Zi≤1/(1−γ)Z_i\leq 1/(1-\gamma) for γ<1\gamma<1.

Our original evaluation covers tasks with different episode-length settings. To directly assess this scoring choice, we additionally compared Uniform, REAP with Uniform base, and REAP using si/Zis_i/Z_i on four sparse-reward and four dense-reward tasks. The table reports AUC as mean ± standard deviation.

Reward typeTaskUniformREAPREAP + normalization
SparseLiftPegUpright0.00 ± 0.000.72 ± 0.110.70 ± 0.09
SparsePushCube0.45 ± 0.040.87 ± 0.020.82 ± 0.10
SparsePullCube0.63 ± 0.020.88 ± 0.020.87 ± 0.03
SparsePlaceSphere0.00 ± 0.000.38 ± 0.070.38 ± 0.06
DenseHopperStand0.24 ± 0.120.54 ± 0.170.53 ± 0.16
DenseGo2 Walk0.27 ± 0.020.39 ± 0.050.33 ± 0.09
DenseFranka Lift0.03 ± 0.090.74 ± 0.040.69 ± 0.05
DenseT1 Rough0.42 ± 0.010.53 ± 0.010.51 ± 0.06

Normalized REAP retains higher mean AUC than Uniform on all eight tasks. It yields lower mean AUC than the original REAP on seven tasks and the same reported mean on one. These results show that REAP’s gains persist under both scoring rules, supporting robustness to effective-length normalization and our choice of cumulative scoring as a practical design. We thank the reviewer for motivating this comparison, which further clarifies the empirical basis for this choice.

AuthorsOpenReview ↗

Response to Reviewer cKyy (2/2)

W3 / Q3: Warmup transitions

Once rrefr_{\mathrm{ref}} is fixed at the end of warmup, we compute centered return-to-go scores for the stored transitions from episodes that have already ended. Positive-score transitions then become eligible for the return branch, while all transitions remain accessible through the base branch.

Transitions in unfinished episodes are sampled only through the base branch until their episodes end. They are then scored using the same fixed reference. Since parallel episodes can finish at different times, this allows learning to proceed without waiting for all ongoing episodes to finish, while computing return priorities from completed trajectories.

We will clarify this procedure in the revised manuscript and release our code with the warmup and scoring logic documented.


W4 / Q4: Comparisons with related methods

Following your suggestion, we added comparisons inspired by AE-DDPG [1] and ISAC [2]. We also evaluated advantage-guided replay and self-imitation. All four mechanisms are implemented within the same FlashSAC framework:

  • Episodic replay: Inspired by AE-DDPG [1], this variant stores episodes matching or exceeding the best observed return in a separate memory and mixes samples from this memory with ordinary replay.
  • Episode-return Top-K: Inspired by ISAC [2], this variant ranks 2B2B uniformly sampled candidate transitions from completed episodes by their episode returns and retains the top BB.
  • Advantage-guided replay: Adapting the positive-advantage prioritization in SIL [3], this variant samples transitions using cached positive Q-based advantages while retaining the standard FlashSAC losses.
  • Self-imitation: Inspired by the imitation objective in SIL [3], this variant adds an objective weighted by positive Q-based advantages to imitate stored actions under uniform replay.

Both advantage-based variants use a Q-based advantage estimate following AWAC [4], computed as the stored action’s Q-value minus the average Q-value of four actions sampled from the current policy.

Implementation details and learning curves are provided in Appendix F and Figure 20. The table reports normalized success-rate AUC as mean ± standard deviation on four sparse-reward tasks. FlashSAC and REAP use five runs each; each added baseline uses three.

TaskFlashSACEpisodic replayEpisode-return Top-KAdvantage-guided replaySelf-imitationREAP
LiftPegUpright0.00 ± 0.000.10 ± 0.040.00 ± 0.000.00 ± 0.010.00 ± 0.000.72 ± 0.11
PokeCube0.00 ± 0.000.02 ± 0.030.00 ± 0.000.00 ± 0.000.00 ± 0.000.12 ± 0.12
PlaceSphere0.00 ± 0.000.01 ± 0.020.00 ± 0.000.00 ± 0.000.00 ± 0.000.38 ± 0.07
PickCube0.00 ± 0.000.00 ± 0.000.00 ± 0.000.01 ± 0.000.00 ± 0.000.21 ± 0.11

REAP achieves the highest mean AUC and final mean success rate on all four tasks. Episodic replay improves learning on several tasks, while REAP reaches higher success rates earlier, as shown in Figure 20. REAP scores individual transitions by their centered return-to-go, allowing priorities to vary within the same episode while using later rewards to guide replay of earlier transitions. These scores come directly from observed outcomes, so favorable experience can receive additional replay without first being assigned a positive advantage by the critic. These properties may help explain REAP’s stronger performance when successful experience is scarce.

[1] Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments.

[2] Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.

[3] Self-Imitation Learning.

[4] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets.

Thank you for the detailed responses.

Reviewer_xCPFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a clear bottleneck in massively parallel RL: large-scale parallel interaction can make rare successful trajectories available, yet the policy may still require a very long time to learn how to reproduce those behaviors reliably.

The authors call this phenomenon the discovery–mastery gap: successful experience is already present in replay, but deterministic policy performance improves much later or may fail to improve within the entire training budget.

The paper first demonstrates this gap empirically, then performs controlled replay interventions showing that increasing the replay share of successful experience can significantly accelerate mastery while holding collection and update budgets fixed.
The method is evaluated across 48 control tasks in ManiSkill, Genesis, IsaacLab, and MuJoCo Playground, under sparse and dense rewards, and with both SAC-like and TD3-style training.

Critical strengths
  1. The scientific narrative is unusually clean.

    The paper first identifies a phenomenon, then performs an intervention, and only afterward proposes a method. The discovery–mastery gap therefore feels like a genuine empirical finding rather than terminology introduced to justify an algorithm.

  2. The intervention experiments provide strong causal evidence.

    By restoring the same learner and replay buffer and changing only the replay allocation, while keeping the collection and update budgets fixed, the authors provide convincing evidence that replay allocation itself is a bottleneck.

  3. The method is simple and well matched to the problem.

    Return-to-go is cheap to compute, fixed after the episode ends, and does not require repeatedly recomputing TD-error priorities.

  4. The experimental breadth is strong.

    Forty-eight tasks across multiple simulators, sparse and dense rewards, and multiple off-policy algorithms provide good support for generality.

  5. Budget analysis is extensive.

    The paper studies batch size, update count, and parallel-environment scaling, rather than only one default regime.

  6. The ablations are well targeted.

    Alternative replay scores, importance correction, return transformations, and the mixing coefficient are all examined systematically.

Critical weaknesses
  1. The algorithmic novelty is moderate.

    Prioritizing high-return or successful experience is closely related to prior work on prioritized replay, self-imitation, and episodic replay. The strongest novelty of the paper lies more in the diagnosis of the discovery–mastery gap than in the replay rule itself.

  2. REAP intentionally introduces sampling bias without importance correction.

    The paper empirically finds that importance correction hurts performance, which is reasonable from an engineering perspective. However, the resulting training distribution is deliberately shifted toward high-return experience, which raises questions about the underlying objective.

  3. High return is not always equivalent to useful or generalizable experience.

    In settings with reward hacking, noisy rewards, or local shortcuts, REAP may over-amplify trajectories that are high-return but undesirable.

  4. The fixed reward reference �0��0� may not be robust in all settings.

    A random-policy mean reward is simple, but it may become less meaningful in nonstationary curricula or tasks with strong episode-length effects.

  5. The empirical scope is broad within simulated control, but narrow outside it.

    The paper would be even stronger with one non-control domain to support broader claims about large-scale RL.

Questions for authors
  1. How robust is REAP to noisy or exploitable reward functions?
  2. Why is a fixed random-policy reward reference preferable to alternatives such as a running quantile or median return?
  3. Could REAP reinforce a suboptimal local strategy if that strategy initially produces the highest returns?
  4. Can the authors analyze critic targets or gradient statistics before and after replay reallocation to explain the acceleration mechanistically?
  5. How complementary is REAP to HER, self-imitation, or episodic-memory methods?
  6. Does the advantage of REAP vanish at very high replay ratios?
5 comments
AuthorsOpenReview ↗

Response to Reviewer xCPF (1/5)

We thank the reviewer for the thoughtful assessment and constructive questions. We appreciate your recognition of the empirical findings and controlled experiments, and address your concerns below with additional experiments and clarifications.


W1: Contribution and design value

Our contribution centers on identifying and addressing a bottleneck in massively parallel RL: successful experience can become available long before the policy learns to reproduce it reliably. Through controlled interventions that restore the same learner and replay buffer and keep interaction and update budgets fixed, we show that changing replay allocation can substantially accelerate mastery. These findings highlight the importance of experience utilization as parallel simulation makes successful trajectories increasingly available.

REAP builds on the established idea of prioritizing favorable experience. We deliberately pursue a simple, reusable design that requires minimal changes to existing training pipelines. REAP changes replay allocation while retaining the underlying learner and update rules. Its scores are computed directly from observed returns when an episode ends and then kept fixed, keeping priority maintenance inexpensive. Evaluations across tasks, reward settings, and both SAC-based and TD3-based learners support the practical value and applicability of this design. We discuss the additional comparisons with related methods in our response to Q5.


W2: Sampling bias and the optimization objective

REAP preserves the task reward and the form of the learner’s per-sample actor and critic losses, while intentionally changing how experience is weighted during optimization. Its purpose is to allocate a limited replay budget toward favorable experience that has already been discovered. This allocation is defined in Section 3.2, Eq. (2) and illustrated in Figure 5: positive-return transitions constitute only 2.40% of the buffer but receive over half of the sampling probability under both REAP variants. Retaining this additional emphasis during learning is central to the design.

Non-uniform replay does not inherently require correction toward uniform sampling. Whether importance correction is appropriate depends on which reference distribution’s expected updates one intends to preserve. TGS [1], for example, uses non-uniform replay without importance correction to emphasize recent experience. REAP instead emphasizes favorable observed outcomes. In our ablation, PER-style importance weighting moves updates toward the corresponding base distribution, Uniform or TGS, thereby reducing the additional emphasis introduced by return-guided sampling.

The original results in Section 4.3, Figure 9(b) and Appendix E.1, Table 5 show that this weighting reduces mean normalized AUC for both REAP variants. To clarify its effect on learning, we have added success-rate and critic-loss curves in Appendix E.1, Figure 17. The table below summarizes mean normalized success-rate AUC over five seeds for the two tasks shown there.

TaskBase samplerBaseREAPREAP + IS
LiftPegUprightUniform0.000.72<0.01
LiftPegUprightTGS0.000.79<0.01
PushCubeUniform0.450.870.47
PushCubeTGS0.510.870.57

With importance weighting, learning becomes much closer to the corresponding base sampler. On LiftPegUpright, nearly all of REAP’s improvement disappears. On PushCube, the agent still learns the task, but learning slows substantially and most of the AUC advantage is lost. The recorded critic losses in Figure 17 show no obvious sustained divergence. These observations are consistent with importance weighting weakening the additional learning emphasis introduced by REAP, and support retaining that emphasis under the evaluated training budgets.

[1] When Does Non-Uniform Replay Matter in Reinforcement Learning?

AuthorsOpenReview ↗

Response to Reviewer xCPF (2/5)

W3: The role and limitations of return-guided replay

High return alone does not establish learning value or generalizability. REAP uses observed return as a simple, task-related signal for allocating replay. Its motivation comes from the discovery–mastery gap: favorable experience is already available, but the policy has not learned to reproduce it reliably. The controlled interventions in Figure 4 show that increasing replay of successful experience accelerates mastery from the same starting learner and buffer, under matched collection and update budgets. This provides empirical support for prioritizing such experience in the setting we study.

Our sparse-reward experiments further examine this principle using rewards tied directly to task success, without intermediate reward shaping. The improvements in Figure 6 show that REAP’s benefits extend to this setting. REAP also retains a base sampling component, as defined in Section 3.2, Eq. (2), so lower-return and unsuccessful experience remain available for learning.

When rewards systematically favor undesirable behavior, return-guided replay can also reinforce that behavior. Identifying or correcting reward misspecification is outside the scope of REAP. For stochastic reward noise, our additional experiments show that REAP retains its learning-efficiency advantage at the tested noise levels; the settings and results are provided in Q1.


W4: Fixed reward reference and episode length

The reference in Section 3.1, Eq. (1) accounts for the reward background observed during random-policy warmup. Keeping it fixed gives experience collected at different training stages a common scoring rule, consistent with computing each score once and retaining it afterward. The reference ablations in Q2 show that several fixed choices preserve gains over the baseline, with the warmup mean remaining competitive across tasks. These results support it as a practical default without task-specific reference tuning.

The score does depend on the remaining episode length because it measures cumulative discounted reward above the reference. Discounting limits this dependence but does not eliminate it. To examine its effect, we compare the original score with a variant that divides it by the effective discounted remaining length, Zi=∑k=0Li−1γkZ_i=\sum_{k=0}^{L_i-1}\gamma^k. This changes the score from cumulative discounted excess reward to discounted average excess reward per step. Both REAP variants use a Uniform base and λ=0.5\lambda=0.5, with the remaining training settings unchanged.

The table reports AUC as mean ± sample standard deviation over five seeds.

Reward typeTaskUniformREAPREAP + length normalization
SparseLiftPegUpright0.00 ± 0.000.72 ± 0.110.70 ± 0.09
SparsePushCube0.45 ± 0.040.87 ± 0.020.82 ± 0.10
SparsePullCube0.63 ± 0.020.88 ± 0.020.87 ± 0.03
SparsePlaceSphere0.00 ± 0.000.38 ± 0.070.38 ± 0.06
DenseHopperStand0.24 ± 0.120.54 ± 0.170.53 ± 0.16
DenseGo2 Walk0.27 ± 0.020.39 ± 0.050.33 ± 0.09
DenseFranka Lift0.03 ± 0.090.74 ± 0.040.69 ± 0.05
DenseT1 Rough0.42 ± 0.010.53 ± 0.010.51 ± 0.06

Length normalization retains higher mean AUC than Uniform on all eight tasks. Compared with the original REAP score, mean AUC decreases on seven tasks and matches on one at the reported precision. Thus, the learning gains persist under this normalization, while the results support retaining cumulative scoring as our default.

For nonstationary curricula, the gains across multiple fixed references and reward settings motivate exploring REAP beyond the current evaluation. When task or reward backgrounds change during training, an important direction for future work is to study whether reference recalibration is needed and how to maintain consistent scoring while preserving the method’s low maintenance cost.


W5: Evaluation in arcade games

We have extended our evaluation beyond robotic manipulation and locomotion to four MiniArcade games from MMBench: Spaceship, Pong, Rocket Collect, and Air Hockey.

MMBench: https://github.com/nicklashansen/newt

We compare FlashSAC and REAP over the first one million environment steps. The table reports AUC as mean ± standard deviation across five random seeds.

TaskFlashSAC ↑REAP ↑
Spaceship0.33 ± 0.070.65 ± 0.05
Pong0.42 ± 0.090.59 ± 0.03
Rocket Collect0.22 ± 0.040.26 ± 0.07
Air Hockey0.35 ± 0.040.50 ± 0.02

REAP achieves higher mean AUC on all four games. These results provide additional evidence that return-guided replay can improve learning efficiency beyond the robotics benchmarks studied in the main paper.

AuthorsOpenReview ↗

Response to Reviewer xCPF (3/5)

Q1: Robustness to noisy and exploitable rewards

We evaluate reward-noise sensitivity on PlaceSphere with dense rewards and PushCube with sparse rewards, comparing FlashSAC with REAP using a TGS base. We add independent Gaussian noise to each collected training reward, including during warmup: r~t=rt+ϵt\tilde r_t=r_t+\epsilon_t, where ϵt∼N(0,σ2)\epsilon_t\sim\mathcal N(0,\sigma^2). We test noise standard deviations of 0, 0.05, and 0.10. The perturbed rewards are used for learning updates, REAP’s reward reference, and replay scores. Evaluation remains noise-free.

Each setting uses five random seeds and approximately 20 million environment steps. The table reports AUC as mean ± sample standard deviation.

EnvironmentMethodσ=0.00\sigma=0.00σ=0.05\sigma=0.05σ=0.10\sigma=0.10
PlaceSphere (Dense)FlashSAC0.36 ± 0.120.31 ± 0.210.25 ± 0.29
PlaceSphere (Dense)REAP0.63 ± 0.060.60 ± 0.110.57 ± 0.18
PushCube (Sparse)FlashSAC0.45 ± 0.040.40 ± 0.120.33 ± 0.18
PushCube (Sparse)REAP0.87 ± 0.030.83 ± 0.100.79 ± 0.14

Both methods show lower mean AUC as noise increases, while REAP retains higher mean AUC on both tasks. From σ=0\sigma=0 to σ=0.10\sigma=0.10, REAP’s mean AUC decreases by 0.06 on PlaceSphere and 0.08 on PushCube, compared with 0.11 and 0.12 for FlashSAC. These results show that REAP preserves its learning-efficiency advantage under the tested reward perturbations.

If a reward function systematically assigns high returns to undesirable behavior, REAP may also reinforce that behavior. Identifying or correcting such reward misspecification is outside the scope of this work. Our emphasis on success-based sparse rewards also reduces reliance on intermediate reward shaping and ties the reward signal more directly to task completion.


Q2: Choice of the reward reference

We use the random-policy warmup mean as a simple reference for the task’s reward background. As described in Section 3.1, Eq. (1), centering helps distinguish favorable outcomes when random behavior already receives nonzero rewards. It affects replay scores only; the learner still uses the original rewards. Keeping the reference fixed gives experience collected at different training stages a common scoring rule, consistent with REAP’s design of computing each score once when a trajectory ends.

We evaluate this choice on six tasks using five seeds per setting. We compare the default fixed mean with Zero, fixed warmup Q25 and Q75, and an online EMA. Q25 and Q75 are quantiles of warmup per-step rewards. EMA starts from the warmup mean and uses a half-life of 500,000 environment transitions. Its updated reference applies to newly completed trajectories, while stored scores remain frozen. All REAP variants use a Uniform base and λ=0.5\lambda=0.5.

The table reports mean ± sample standard deviation of AUC. Manipulation tasks use success rate, and Humanoid tasks use original episode return.

TaskFlashSACREAP (default)REAP (Zero)REAP (Q25)REAP (Q75)REAP (EMA)
LiftPegUpright (Sparse)0.00 ± 0.000.72 ± 0.110.71 ± 0.120.72 ± 0.080.73 ± 0.100.52 ± 0.26
PushCube (Sparse)0.45 ± 0.040.87 ± 0.020.86 ± 0.030.87 ± 0.020.87 ± 0.010.57 ± 0.07
PlaceSphere (Dense)0.36 ± 0.120.64 ± 0.020.49 ± 0.050.57 ± 0.080.60 ± 0.060.46 ± 0.17
PegInsertionSide (Dense)0.43 ± 0.080.54 ± 0.030.48 ± 0.040.56 ± 0.020.52 ± 0.050.44 ± 0.10
MS-HumanoidWalk (Dense)518 ± 2614 ± 53556 ± 53612 ± 49607 ± 60483 ± 72
MS-HumanoidStand (Dense)365 ± 170535 ± 59523 ± 58530 ± 61530 ± 62492 ± 74

Zero retains gains over FlashSAC on all six tasks, while the default achieves higher mean AUC on all four dense tasks. This supports using centering to accommodate nonzero reward backgrounds. On the sparse tasks, the warmup reference is zero or nearly zero, and the fixed-reference variants perform similarly.

Both fixed quantiles also retain higher mean AUC than FlashSAC across the six tasks. The default is not always the highest-performing choice, but remains competitive. These results support the warmup mean as a practical default without requiring a precisely tuned reference for each task.

The tested EMA produces lower mean AUC and greater cross-seed variability than the default on all six tasks. One possible explanation is that, with historical scores frozen, a changing reference makes scores assigned at different training stages less comparable. The results therefore support retaining a fixed reference within the current scoring design.

Reviewer_PrupOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a gap between discovery and mastery in massively parallel RL: rare successes are easy to collect at scale, yet policies often fail to reproduce them reliably within the budget. The authors document this with diagnostics on three quantities (stochastic success rate, success fraction in the buffer, deterministic success rate) and test causality with an intervention study that restores the learner and buffer at five checkpoints per task and replays half of each batch uniformly from successful episodes, with budgets unchanged. Motivated by this, they propose REAP, which mixes a base sampler (uniform replay or TGS) with sampling proportional to the positive part of a centered return to go. Scores are computed once per episode with a reward reference from a fixed random policy and never updated, and no importance correction is applied. With one setting (lambda = 1/2) across all tasks, REAP learns on four of eight ManiSkill tasks with sparse rewards where uniform replay and TGS stay near zero and PER improves only slightly, and improves normalized AUC by roughly 0.11 to 0.14 over the corresponding baselines on the remaining 40 tasks under FlashSAC. A TD3 study on eight tasks shows the same trend. At half the replay budget REAP still beats the baselines at their full budget, at 6 to 8 percent wall clock overhead.

Critical strengths
  1. The combination of careful diagnostics and a causal intervention study is the main reason I lean positive. The diagnostics in Figure 3 track three quantities (stochastic success rate, success fraction in the buffer, deterministic success rate) and document the gap cleanly, and the checkpoint intervention in Section 2.2 is a clean causal test that is rarely done in this area: same learner, same buffer, only the sampling distribution changes, with budgets matched. The fact that replay reallocation alone can rescue training after millions of steps without success is convincing support for the paper's central claim.
  2. The empirical work is broad and the statistics are done properly. The main study covers 48 tasks across four simulators under FlashSAC, with TD3 evaluated on eight tasks as a transfer check. The main comparisons use five seeds and report bootstrap confidence intervals, under one fixed lambda without per task tuning. The positive part versus absolute value ablation additionally reports a probability of improvement analysis. The appendix reports curves for every task together with the normalization references. The budget and scaling analyses (halving B or U, a 4x batch size, and N from 512 to 2048 under two update schedules) go well beyond what replay papers usually provide. The method itself is cheap (about 6 to 8 percent overhead) and still beats the baselines at their full budget when given only half the replay budget, which makes it easy to adopt.
Critical weaknesses
  1. The paper never compares against the closest success focused baselines, methods that learn from past successes. Self imitation learning (Oh et al., 2018) and methods that weight experience by advantage, such as AWR and AWAC, also exploit experience with high returns, although they modify the policy update rather than only the sampling distribution. Running them, or a replay level variant of the same idea, would clarify how much of the gain comes from emphasizing successful experience in general rather than from REAP's specific sampling rule. The four sparse tasks where uniform replay fails to learn are exactly where such baselines matter most. Without them it is hard to tell whether REAP solves these tasks or whether stronger baselines that focus on successful experience would solve them too. This matters because the mechanism itself is a combination of known ingredients, as replay scores based on returns go back to at least Zhang et al. (2019) and Tafazzol et al. (2021) and self imitation to Oh et al. (2018). Running these baselines is what would establish the actual increment of REAP. I consider this fixable with additional experiments, but it directly affects how much contribution I can credit, and it is my main hesitation.
  2. Two central design choices are not ablated and both may degrade as training proceeds. First, r_ref is the mean reward per step of a random policy and is held fixed for the whole run. Once the policy improves, the average reward moves far above r_ref, and I would expect the centered return to go to turn positive for nearly every transition on dense tasks, at which point the positive part stops selecting anything and REAP reduces to plain return weighting over almost the whole buffer. The paper ablates the scoring signal, lambda, and positive part versus absolute value, but never the reference itself, and it never reports how the fraction of transitions with positive scores evolves over training. Second, scores are frozen at collection time, so trajectories with high returns collected long ago keep receiving replay after the policy has surpassed them, and no refresh or decay mechanism is studied. Related to this, the decision to skip importance correction is argued from an ablation where adding IS hurts both variants (on LiftPegUpright, AUC drops from 0.79 to 0.001) and from the precedent that TGS also omits correction. The LiftPegUpright drop looks more like an optimization or implementation instability than a principled argument, and the bias introduced by uncorrected sampling is never quantified.
Questions for authors
  1. Could you add comparisons with self imitation (Oh et al., 2018) or a simple replay baseline that weights experience by advantage on the four sparse tasks where uniform replay fails? How much of REAP's gain on these tasks would those baselines also obtain? This would materially change my assessment of the contribution.
  2. How sensitive is REAP to the choice of r_ref? Please report an ablation over r_ref fixed at 0, an online running mean, and a fixed quantile, together with a curve of the fraction of buffer transitions whose centered return to go is positive over the course of training. Does the positive part still single out a small subset late in training on dense tasks?
  3. In Table 5, adding importance weighting drops LiftPegUpright from 0.79 to 0.001. Is this a fundamental problem with applying correction under this sampling scheme, or an instability in the implementation? Since the uncorrected updates are biased, even a rough quantification of that bias would help justify leaving the correction out.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Prup (1/3)

We thank the reviewer for the careful and constructive review, and for recognizing the value of our diagnostics, replay interventions, and broad empirical evaluation. We address your comments and questions below.


W1 / Q1: Comparisons with advantage-guided replay and self-imitation

Following your suggestion, we added advantage-guided replay and self-imitation comparisons on the four sparse-reward tasks. Both use the same Q-based advantage definition, following AWAC [1]: the stored action’s estimated Q-value minus the average estimated Q-value of four actions sampled from the current policy. We use the positive part of this estimate in two ways:

  • Advantage-guided replay: Adapting the positive-advantage prioritization in SIL [2], this variant samples transitions using cached advantage weights while retaining the standard FlashSAC losses.
  • Self-imitation: Inspired by SIL’s imitation objective [2], this variant adds an auxiliary objective that increases the likelihood of stored actions, weighted by their positive estimated advantages, while retaining uniform replay.

We also included episodic replay inspired by AE-DDPG [3], which maintains a separate memory of high-return episodes, and Episode-return Top-K inspired by ISAC [4], which selects transitions from uniformly sampled candidates according to episode return. All four mechanisms are implemented within FlashSAC.

Implementation details and learning curves are provided in Appendix F and Figure 20. The table reports normalized success-rate AUC as mean ± standard deviation. FlashSAC and REAP use five original runs each; each added baseline uses three.

TaskFlashSACAdvantage-guided replaySelf-imitationEpisodic replayEpisode-return Top-KREAP
LiftPegUpright0.00 ± 0.000.00 ± 0.010.00 ± 0.000.10 ± 0.040.00 ± 0.000.72 ± 0.11
PokeCube0.00 ± 0.000.00 ± 0.000.00 ± 0.000.02 ± 0.030.00 ± 0.000.12 ± 0.12
PlaceSphere0.00 ± 0.000.00 ± 0.000.00 ± 0.000.01 ± 0.020.00 ± 0.000.38 ± 0.07
PickCube0.00 ± 0.000.01 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.21 ± 0.11

Under these settings, advantage-guided replay and self-imitation obtain little improvement over FlashSAC and do not recover most of REAP’s AUC gains. Episodic replay improves learning on several tasks, but REAP achieves the highest mean AUC on all four.

Two properties may help explain these results. First, REAP assigns priorities directly from observed centered return-to-go, allowing favorable experience to receive additional replay without first being assigned a positive advantage by the current critic. This can be useful when successful experience is scarce and its value is still being learned. Second, REAP directly increases how often such experience enters training, benefiting both critic and actor updates. The tested self-imitation variant instead adds supervision to uniformly sampled batches, leaving the frequency of sampling rare experience unchanged.

REAP computes priorities once from existing rewards, requires no task-specific success labels, and preserves the form of the learner’s per-sample losses without introducing an auxiliary objective.

References

[1] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets.

[2] Self-Imitation Learning.

[3] Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments.

[4] Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.

AuthorsOpenReview ↗

Response to Reviewer Prup (2/3)

W2 / Q2: Motivation and sensitivity of the fixed reward reference

REAP uses a fixed reward reference to score observed outcomes consistently across training. In dense tasks, even random behavior can receive positive rewards. Centering reduces this background contribution to replay scores, using the mean per-step reward from random warmup without an additional value estimator or task-specific tuning. The learner still uses the original rewards.

Keeping the reference fixed lets experience collected at different stages follow the same scoring rule, consistent with computing each score once at episode completion. It calibrates the reward background rather than tracking the current policy’s performance.

We compare the default fixed warmup mean with Zero, fixed warmup Q25 and Q75, and an online EMA on two sparse and four dense tasks. Q25 and Q75 are quantiles of per-step warmup rewards. EMA starts from the warmup mean with a half-life of 500,000 transitions and affects newly completed trajectories; stored scores remain frozen. All variants use a uniform base and a mixing coefficient of 0.5.

The table below reports the fixed reference values averaged over five warmup runs, rounded for readability.

TaskWarmup mean (default)Q25Q75
LiftPegUpright (Sparse)000
PushCube (Sparse)9.96e−500
PlaceSphere (Dense)0.040.020.05
PegInsertionSide (Dense)0.020.010.03
MS-HumanoidWalk (Dense)0.500.500.50
MS-HumanoidStand (Dense)9.40e−300

Performance is reported as AUC, mean ± sample SD over five runs, using success rate for manipulation tasks and episode return for Humanoid tasks.

TaskFlashSACREAP (default)REAP (Zero)REAP (Q25)REAP (Q75)REAP (EMA)
LiftPegUpright (Sparse)0.00 ± 0.000.72 ± 0.110.71 ± 0.120.72 ± 0.080.73 ± 0.100.52 ± 0.26
PushCube (Sparse)0.45 ± 0.040.87 ± 0.020.86 ± 0.030.87 ± 0.020.87 ± 0.010.57 ± 0.07
PlaceSphere (Dense)0.36 ± 0.120.64 ± 0.020.49 ± 0.050.57 ± 0.080.60 ± 0.060.46 ± 0.17
PegInsertionSide (Dense)0.43 ± 0.080.54 ± 0.030.48 ± 0.040.56 ± 0.020.52 ± 0.050.44 ± 0.10
MS-HumanoidWalk (Dense)518 ± 2614 ± 53556 ± 53612 ± 49607 ± 60483 ± 72
MS-HumanoidStand (Dense)365 ± 170535 ± 59523 ± 58530 ± 61530 ± 62492 ± 74

The results support three observations:

  • Centering adds benefits on dense tasks. Zero already improves over FlashSAC on all six tasks, but the default achieves higher mean AUC on all four dense tasks, especially PlaceSphere (0.64 versus 0.49) and HumanoidWalk (614 versus 556). Sparse-task results remain similar, consistent with their near-zero references.

  • The tested fixed references preserve the gains. Q25 and Q75 both outperform FlashSAC in mean AUC on all six tasks. PlaceSphere and PegInsertionSide provide the clearest sensitivity tests because their references differ substantially. The default remains competitive without being universally best, supporting the warmup mean as a practical common choice.

  • The tested EMA performs worse within the current scoring scheme. It yields lower mean AUC and larger cross-run SD than the default on all six tasks. One possible explanation is that a changing reference combined with frozen historical scores reduces consistency across collection times: better later outcomes can receive lower scores under a higher reference.

Finally, more transitions may exceed the fixed reference as learning progresses. The positive-part filter then becomes less selective, while relative score magnitudes continue to determine replay allocation. Maintaining a small positive-score set throughout training is not a requirement of REAP’s design.

We keep stored scores fixed to avoid repeated rescoring and its maintenance cost. These scores summarize observed outcomes rather than estimate each transition’s current learning value. Fixed scores do not imply fixed sampling probabilities: their relative weights change with the buffer’s composition, and older transitions are eventually overwritten. Previously useful experience may therefore continue to receive replay after the policy improves. This is a deliberate tradeoff between low maintenance cost and adapting priorities to the current learner; we have not evaluated score refresh or decay.

AuthorsOpenReview ↗

Response to Reviewer Prup (3/3)

Q3: Importance correction and sampling bias

To clarify the large AUC drop, Figure 17 in Appendix E.1 shows success-rate and critic-loss curves for LiftPegUpright and PushCube. The table below reports mean AUC over five seeds.

TaskBase samplerBaseREAPREAP + IS
LiftPegUprightUniform0.0000.7240.003
LiftPegUprightTGS0.0000.7910.001
PushCubeUniform0.4490.8720.473
PushCubeTGS0.5080.8710.572

Adding IS brings performance much closer to the corresponding base sampler. On LiftPegUpright, the corrected variants show little effective learning through most of training, resembling the unsuccessful bases. On PushCube, they still reach high final success, but learning slows substantially and most of REAP’s AUC advantage disappears. The recorded critic losses show no obvious sustained divergence, although these curves alone cannot exclude all optimization or implementation effects.

Our interpretation follows REAP’s design: it intentionally increases the contribution of favorable observed experience under a limited replay budget. The IS ablation keeps sampling unchanged while moving updates toward the corresponding base distribution, Uniform or TGS. Thus, experience sampled more frequently through return guidance receives less weight per update. The observed loss of learning gains is consistent with this correction weakening REAP’s additional emphasis. It does not establish a fundamental incompatibility between IS and REAP.

Figure 5 quantifies the sampling distribution change. In the PullCube buffer at 5M environment steps, positive-return transitions constitute only 2.40% of the buffer but receive 51.20% of the sampling probability under REAP with a uniform base. This measures the magnitude of the intended replay reallocation, not gradient bias or value-estimation error.

Together, these results support retaining the additional experience weighting under the evaluated training budgets. We do not claim that the uncorrected updates are unbiased relative to the base distribution.

Thank you for the revision and the detailed responses. The new experiments address my concerns. I will maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.