- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies replay allocation in massively parallel off-policy reinforcement learning and identifies a “discovery–mastery gap,” where successful trajectories may already exist in replay while the policy still fails to reproduce them reliably. The authors propose REAP, which increases replay probability for transitions with positive centered return-to-go while retaining a base sampler. Experiments across 48 control tasks show improved learning efficiency over uniform replay, TGS, and PER. Presentation
Critical strengths
1.The paper isolates an interesting mismatch between discovering successful behavior and learning to reproduce it, and uses replay-switch interventions to test the role of replay allocation. 2. REAP is simple to integrate with an off-policy learner. Its scores are computed from observed rewards when episodes end, without repeatedly updating priorities from critic estimates. 3. The evaluation is broad, covering 48 tasks across four benchmarks. The paper also reports results with TD3, wall-clock time, and replay-budget ablations.
Critical weaknesses
- Unclear distinction between discovery and mastery. The paper uses stochastic evaluation to measure behavior discovery and deterministic evaluation to measure policy mastery. However, the different action-selection strategies may contribute to the observed performance gap, making it difficult to determine whether this gap truly reflects a lack of policy mastery.
2.Effect of episode length on replay priorities. The centered return-to-go depends on the remaining episode length, but the paper does not analyze how this affects replay priorities across episodes of different lengths.
- Insufficient implementation details. The paper does not clearly explain how warmup transitions are handled after the reference reward is established. This makes the replay sampling procedure difficult to fully understand and reproduce.
4.Limited comparison with existing replay strategies. Although REAP is compared with Uniform, TGS, and PER, the experiments do not establish its advantages over other episodic or return-based replay mechanisms explored in prior work.
Questions for authors
1.In Figure 3, the authors use deterministic evaluation to measure policy mastery and stochastic evaluation to indicate successful behavior discovery. Could the authors clarify how deterministic actions are selected from the learned policy (e.g., using the mean action)? Furthermore, why is deterministic success rate considered an appropriate measure of mastery, rather than evaluating the reliability of the stochastic policy itself? 2.In Section 3.1, the centered return-to-go depends on the remaining episode length (L_i). How does this affect the sampling probabilities of transitions from episodes of different lengths? Have the authors considered this effect in their experiments? 3.How are warmup transitions handled in REAP, given that is determined only after warmup? Are they scored retrospectively or sampled only through the base distribution? 4.Several prior studies have explored improved experience replay strategies for continuous-control RL, such as AE-DDPG [1] and ISAC [2]. Could the authors provide further comparisons with these or other relevant replay methods to better evaluate the effectiveness of REAP?
[1]Asynchronous episodic deep deterministic policy gradient: Toward continuous control in computationally complex environments. [2]Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.
3 comments
Response to Reviewer cKyy (2/2)
W3 / Q3: Warmup transitions
Once is fixed at the end of warmup, we compute centered return-to-go scores for the stored transitions from episodes that have already ended. Positive-score transitions then become eligible for the return branch, while all transitions remain accessible through the base branch.
Transitions in unfinished episodes are sampled only through the base branch until their episodes end. They are then scored using the same fixed reference. Since parallel episodes can finish at different times, this allows learning to proceed without waiting for all ongoing episodes to finish, while computing return priorities from completed trajectories.
We will clarify this procedure in the revised manuscript and release our code with the warmup and scoring logic documented.
W4 / Q4: Comparisons with related methods
Following your suggestion, we added comparisons inspired by AE-DDPG [1] and ISAC [2]. We also evaluated advantage-guided replay and self-imitation. All four mechanisms are implemented within the same FlashSAC framework:
- Episodic replay: Inspired by AE-DDPG [1], this variant stores episodes matching or exceeding the best observed return in a separate memory and mixes samples from this memory with ordinary replay.
- Episode-return Top-K: Inspired by ISAC [2], this variant ranks uniformly sampled candidate transitions from completed episodes by their episode returns and retains the top .
- Advantage-guided replay: Adapting the positive-advantage prioritization in SIL [3], this variant samples transitions using cached positive Q-based advantages while retaining the standard FlashSAC losses.
- Self-imitation: Inspired by the imitation objective in SIL [3], this variant adds an objective weighted by positive Q-based advantages to imitate stored actions under uniform replay.
Both advantage-based variants use a Q-based advantage estimate following AWAC [4], computed as the stored action’s Q-value minus the average Q-value of four actions sampled from the current policy.
Implementation details and learning curves are provided in Appendix F and Figure 20. The table reports normalized success-rate AUC as mean ± standard deviation on four sparse-reward tasks. FlashSAC and REAP use five runs each; each added baseline uses three.
| Task | FlashSAC | Episodic replay | Episode-return Top-K | Advantage-guided replay | Self-imitation | REAP |
|---|---|---|---|---|---|---|
| LiftPegUpright | 0.00 ± 0.00 | 0.10 ± 0.04 | 0.00 ± 0.00 | 0.00 ± 0.01 | 0.00 ± 0.00 | 0.72 ± 0.11 |
| PokeCube | 0.00 ± 0.00 | 0.02 ± 0.03 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.12 ± 0.12 |
| PlaceSphere | 0.00 ± 0.00 | 0.01 ± 0.02 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.38 ± 0.07 |
| PickCube | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.01 ± 0.00 | 0.00 ± 0.00 | 0.21 ± 0.11 |
REAP achieves the highest mean AUC and final mean success rate on all four tasks. Episodic replay improves learning on several tasks, while REAP reaches higher success rates earlier, as shown in Figure 20. REAP scores individual transitions by their centered return-to-go, allowing priorities to vary within the same episode while using later rewards to guide replay of earlier transitions. These scores come directly from observed outcomes, so favorable experience can receive additional replay without first being assigned a positive advantage by the critic. These properties may help explain REAP’s stronger performance when successful experience is scarce.
[1] Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex Environments.
[2] Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples With On-Policy Experiences.
[3] Self-Imitation Learning.
[4] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets.
Thank you for the detailed responses.
Response to Reviewer cKyy (1/2)
We thank the reviewer for the careful review and constructive comments. We address the four points below.
W1 / Q1: Discovery and mastery
In Figure 3, successful trajectories occur under stochastic execution, but success remains extremely rare. Across the evaluated checkpoints after initialization, the highest mean stochastic success rates are approximately 0.059% for PlaceSphere within the first 10M steps, 0.234% for PullCube within 5M steps, and 0.012% for LiftPegUpright within 20M steps. Deterministic success rates are zero or near zero over the same respective windows.
We use discovery to indicate that successful trajectories have been generated, and mastery to indicate reliable task completion. A stochastic policy can also demonstrate mastery if it succeeds consistently. Here, the low absolute success rates under both evaluation modes support the conclusion that successful behavior has been discovered but remains unreliable. The difference between action-selection strategies alone would not establish this conclusion.
For deterministic evaluation, we select a=tanh(μθ(s)), where μθ(s) is the Gaussian mean before the tanh transformation. This follows a common SAC practice in which actions are sampled during experience collection and deterministic actions are used for performance evaluation. The resulting success rate measures whether the policy reliably completes the task without action sampling and serves as our operational measure of mastery under this execution protocol.
W2 / Q2: Episode length and replay priorities
Remaining episode length influences sampling through the accumulated centered return. Let Zi=∑k=0Li−1γk and let rˉiγ denote the discounted average reward over the remaining trajectory. The score is si=Zi[rˉiγ−rref]+, and the sampling probability is pi=λsi/∑jsj+(1−λ)pibase. At the same positive average excess reward, a larger Zi therefore gives a transition greater relative weight in the return branch.
We retain cumulative scoring to preserve the total discounted excess reward along each observed continuation, including differences between brief and sustained favorable outcomes. Dividing by Zi changes the criterion to discounted average excess reward. Discounting bounds the effective-length factor by Zi≤1/(1−γ) for γ<1.
Our original evaluation covers tasks with different episode-length settings. To directly assess this scoring choice, we additionally compared Uniform, REAP with Uniform base, and REAP using si/Zi on four sparse-reward and four dense-reward tasks. The table reports AUC as mean ± standard deviation.
Normalized REAP retains higher mean AUC than Uniform on all eight tasks. It yields lower mean AUC than the original REAP on seven tasks and the same reported mean on one. These results show that REAP’s gains persist under both scoring rules, supporting robustness to effective-length normalization and our choice of cumulative scoring as a practical design. We thank the reviewer for motivating this comparison, which further clarifies the empirical basis for this choice.