acceptodds
Under review as a conference paper at ICLR 2027

Freshness-Aware Prioritized Experience Replay for LLM Reinforcement Learning

Abstract

Reinforcement learning (RL) for large language models (LLMs) is expanding from mathematical reasoning to agentic tasks that require costly, multi-turn environment interactions. On-policy training repeatedly collects fresh trajectories and typically does not reuse them across rollout iterations. Experience replay can reuse these interactions, but replay priorities alone may not reflect their relevance as the policy evolves. We introduce FreshPER, a freshness-aware prioritized experience replay method for LLM training. FreshPER combines a base priority with exponential decay in trajectory age to balance priority and recency in the replay distribution. It augments fresh-rollout updates with historical replay while preserving the underlying RL objective and the current-to-behavior policy-ratio correction. Across Sokoban experiments with PPO, GRPO, and a ROLL REINFORCE++ variant, FreshPER improves over priority replay: under PPO, it reaches 43.0% validation success, compared with 33.6% for priority replay and 3.1% for on-policy training. Further results on NQ Search, ScienceWorld, and visual tasks support freshness-aware replay as a practical and effective way to learn more from costly interactions across diverse settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Replay Buffer, LLM post training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jif9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FreshPER, which combines replay priorities with exponential decay in trajectory age for LLM reinforcement learning. It retains behavior-policy probabilities and integrates replay with PPO, GRPO, and a ROLL REINFORCE++ variant. Experiments on Sokoban, search, ScienceWorld, and visual tasks suggest that favoring recent experience can improve learning from a fixed rollout budget.

Critical strengths
  1. The paper addresses a practical bottleneck: expensive environment interactions make trajectory reuse valuable, but useful past experience can become stale as the policy changes. FreshPER is a simple intervention that can be added to existing optimizers. In the matched Sokoban comparison, it improves PPO success from 33.6% to 43.0% over priority replay at the same fresh-trajectory budget (Sec. 4.2).

  2. The evaluation distinguishes fresh-trajectory efficiency from optimization and elapsed-time budgets (Sec. 4.4). The additional budget views make the efficiency claim more informative than a comparison based only on rollout counts.

Critical weaknesses
  1. The NQ Search and ScienceWorld results do not isolate the proposed freshness mechanism. On NQ, Priority and FreshPER differ in buffer capacity, base priority, and learning-rate schedule (Table 5 and Sec. F.2). On ScienceWorld, the learning-rate and importance-weight schedules differ (Sec. F.3). Thus, the large performance gaps could arise from these choices or their interactions. The paper acknowledges this, but the matched Sokoban and supporting VLM results do not establish that freshness explains the gains in the more demanding interactive settings. This limits the main empirical contribution.

  2. The stability of the improvement is not established: each curve represents one training run (Sec. 4.1), so repeated evaluations along a curve do not quantify training-seed variability. This matters particularly for the strongest controlled result. With alpha = 0.6, tau = 500, and a 250-iteration Sokoban horizon, age weighting alone changes the relative sampling weight of equally prioritized trajectories by at most about exp(0.3) = 1.35. A modest sampling tilt can plausibly have a large cumulative effect, but independent repeats are needed to distinguish that effect from training variability. Given the small methodological change, reliable empirical gains carry much of the case for acceptance.

Questions for authors

Please check weaknesses.

1 comment
AuthorsOpenReview ↗

Reply for Reviewer jif9

Thank you for recognizing our practical motivation and multi-budget evaluation framework. We address your two main concerns below:

  • W1 (Isolating freshness on demanding tasks): Beyond Sokoban, our logs contain a strictly controlled single-factor ablation on ScienceWorld (Qwen3-8B, GRPO) keeping source code, buffer (64 8-trajectory groups), base priority, α=0.6\alpha=0.6, IS β\beta schedule, LR, and budgets identical while toggling only enable_age_decay. Both FreshPER runs outperform both Priority runs (18.54/21.88 vs. 18.39/17.63, mean 20.21 vs. 18.01). We will separate this matched matrix from the tuned curves in Section 4.3 and label NQ Search explicitly as a full-configuration comparison.
  • W2 (Stability & magnitude of age tilt): Your calculation (exp⁡(0.3)≈1.35\exp(0.3)\approx 1.35 on Sokoban) aligns directly with our replay-batch diagnostics and reveals a clear dose-response scaling:
  1. On Sokoban (τ=500\tau=500), mean sampled age drops by 4–5%4\text{--}5\% with moderate gains.
  2. On ScienceWorld (τ=10\tau=10, max age ≈30\approx 30), the newest-to-oldest weight ratio reaches exp⁡(0.6×30/10)≈6\exp(0.6 \times 30 / 10) \approx 6, lowering mean sampled age by ≈40%\approx 40\% (22.8–26.7→13.8–14.322.8\text{--}26.7 \to 13.8\text{--}14.3), reducing out-of-clip tokens (0.021–0.022→0.017–0.0190.021\text{--}0.022 \to 0.017\text{--}0.019), and yielding the clearest improvement (+2.20 Dev score).

Furthermore, across all 25 strictly matched pairs in our logs (Sokoban REINFORCE++/GRPO/PPO and ScienceWorld GRPO), FreshPER wins/ties in 21 cases (18/3/4). We will include the full per-run table in the appendix and note in Limitations that these repeats reflect execution-level variance under a shared seed rather than independent random seeds.

Reviewer_fYVxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FreshPER augments ordinary fresh-rollout updates with historical replay. Its sampler multiplies a base priority by an exponential decay in collection age, giving recent trajectories or prompt groups higher relative sampling probability within the eligible buffer. The implementation retains behavior-policy log-probabilities, preserves complete GRPO groups, and recomputes values and GAE for PPO replay. The paper evaluates PPO, GRPO, and a ROLL REINFORCE++ variant on Sokoban, and extends the study to NQ Search, ScienceWorld, and visual tasks. It distinguishes fresh-trajectory efficiency from optimizer-update and elapsed-time budgets, and presents decay sensitivity, sampling-weight ablations, and boundary cases.

Critical strengths
  1. FreshPER combines replay priority with collection-age decay to balance learning value and freshness, with straightforward integration into PPO, GRPO, and a REINFORCE++ variant.

  2. At matched fresh-trajectory budgets on Sokoban, FreshPER improves over Priority by 9.4 percentage points with PPO and 8.6 points with the REINFORCE++ variant. Broader agent and visual experiments suggest further applicability.

Critical weaknesses
  1. Section 4.3 states that NQ Search uses method-specific buffer capacities, base priorities, and learning-rate schedules, while ScienceWorld also uses method-specific optimization settings. Consequently, even the large NQ gap between FreshPER (68.8% EM) and Priority (1.6%) cannot be attributed to freshness alone. Unlike the matched Sokoban comparison, these results leave the independent benefit of age weighting on more realistic agent tasks unclear.

  2. Section 3.2 motivates exponential decay by approximating divergence as locally proportional to rollout age, but the same number of rollout iterations can involve different policy shifts depending on learning rate, replay frequency, and training stage. The paper does not directly measure this relationship.

Questions for authors
  1. Could the authors enlarge the text within the figures, especially Figure 1? Several labels are too small to read comfortably at normal viewing size.

  2. This paper mainly consists of qualitative analysis. Could the authors provide some quantitative analysis?

  3. Could the authors clarify the NQ On-policy discrepancy between Figures 4(a) and 5(b)? EM rises from about 0.27 to 0.4–0.5 in Figure 4(a), whereas the On-policy curve drops near zero after roughly two hours in Figure 5(b). Do these plots use different runs, evaluation ranges, or metrics?

1 comment
AuthorsOpenReview ↗

reply for Reviewer fYVx

Thank you for your thoughtful review and accurate summary of our method. We address each of your points below:

  • W1 (Task-specific settings on NQ and ScienceWorld): While NQ Search evaluates full task-tuned configurations (which we will label explicitly), our logs already contain a strict single-factor ablation on ScienceWorld (Qwen3-8B, GRPO) keeping code, buffer, base priority, α=0.6\alpha=0.6, IS β\beta schedule, LR, and budgets identical while toggling only enable_age_decay. Both FreshPER runs outperform both Priority runs (18.54/21.88 vs. 18.39/17.63, mean 20.21 vs. 18.01). Across all 25 strictly matched pairs in our logs, FreshPER wins/ties in 21 cases (18/3/4). We will separate this matched comparison from the tuned curves in Section 4.3.
  • W2 (Rollout age vs. actual policy shift): Analyzing off-policy diagnostics across all 22 replay runs, we controlled for outer training iteration and found that batch age correlates positively with both out-of-clip token fraction and ∣approx. KL∣\vert{}\text{approx. KL}\vert{} in every single run (partial Spearman ρ=+0.51 to +0.65\rho = +0.51\text{ to }+0.65 on ScienceWorld; +0.13 to +0.52+0.13\text{ to }+0.52 on Sokoban). We will refine Section 3.2 to state this local assumption carefully and reference these measurements.
  • Q1 (Figure text size): We will enlarge all labels—especially in Figures 1 and 2—so they are clearly readable at normal viewing size.
  • Q2 (Quantitative analysis): We will add: (1) the full 25-pair win/tie/loss table; (2) sampled-age shift metrics (≈40%\approx 40\% younger on ScienceWorld, 4–5%4\text{--}5\% on Sokoban); (3) partial correlations between age, out-of-clip fraction, and ∣approx. KL∣\vert{}\text{approx. KL}\vert{}; and (4) existing trajectory/update budget breakdowns.
  • Q3 (NQ On-policy discrepancy between Figs. 4(a) and 5(b)): The two plots use the same EM metric but come from different run batches: Fig. 4(a) shows our main runs, whereas Fig. 5(b) came from an earlier profiling batch whose On-policy run collapsed early. We will remove this cross-batch curve from the main text and report same-batch per-iteration compute breakdowns in the appendix.
Reviewer_ipFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FreshPER improves experience replay for LLM reinforcement learning by combining existing replay priorities with an exponential preference for recently collected trajectories. The method integrates with PPO, GRPO, and a ROLL REINFORCE++ variant to reuse interactions while limiting the influence of older experience. Experiments across text and visual tasks show promising improvements, although several comparisons change additional settings alongside freshness weighting.

Critical strengths

Thank you for submitting your paper. Overall, the concept is clear and addresses a practical problem in LLM agent training. Below are the strengths from my view.

  • Simple and practically useful approach: I particularly like the motivation that a trajectory can retain a high replay priority even after the policy that generated it has become outdated. Combining priority with collection age is a straightforward way to address this without estimating policy divergence for every stored trajectory. The integration also considers relevant implementation details, including preserving behavior-policy log-probabilities, retaining complete groups for GRPO, and recomputing values and GAE for PPO. This makes FreshPER a practical extension of an existing replay pipeline.
  • Promising controlled results and broad experimental coverage: The matched Sokoban comparisons provide useful initial evidence for the contribution of age weighting. Under PPO, FreshPER reaches 43.0% success compared with 33.6% for Priority at the same fresh-trajectory budget, with the declared replay and optimization settings otherwise matched. The REINFORCE++ comparison also shows an improvement, and the visual experiments, particularly MathVista, suggest applicability beyond text-only navigation. I appreciate that you examine actor-update budgets as well as collected trajectories, since additional replay optimization is an important part of the cost.

There are also other points that I appreciated:

  • Reporting limitations: You explicitly include settings where the method offers little benefit or does not outperform the controls.
  • Configuration details: The appendix discloses differences between experimental configurations rather than presenting all comparisons as perfectly controlled.
  • Budget accounting: Counting RLEP’s preparatory data collection toward its acquisition budget is appropriate.
Critical weaknesses

Despite the aforementioned strengths, there are two concerns that prevent me from confidently accepting the main experimental conclusions in their current form.

  • The larger-task comparisons do not isolate freshness weighting: The matched Sokoban experiments are useful, but the NQ Search and ScienceWorld comparisons change several consequential settings simultaneously. On NQ Search, Table 5 gives Priority a 50,000-trajectory buffer and advantage-based priorities, while FreshPER uses a 10,000-trajectory buffer and reward-based priorities. Priority also uses a constant learning rate, whereas FreshPER uses a cosine schedule. On ScienceWorld, the methods again differ in learning-rate schedules and PER correction schedules. These differences are disclosed, but they prevent attributing the performance gaps specifically to freshness weighting. In particular, the NQ comparison of 0.688 EM against 0.016 does not establish that age decay prevents the observed collapse. I would suggest taking the FreshPER configuration on each task and disabling only age decay, while keeping the buffer, priority definition, optimization, and replay budget identical.
  • The wall-clock results appear inconsistent with the main NQ results: I could not reconcile Figure 4(a) with Figure 5(b). In Figure 4(a), the main On-policy curve remains approximately within 25–50% EM and finishes at 48.4%. In the wall-clock plot, the On-policy curve appears near zero over the displayed interval. Appendix C.1 describes a change to the time axis, trailing averages, and small visibility offsets, but these operations do not explain this difference. Are the figures based on different runs, checkpoints, or evaluation metrics? This matters because Figure 5(b) provides the principal evidence that the gains persist under an elapsed-time budget. Please reconcile the figures and provide the underlying checkpoint-level scores and timestamps. Until this is clarified, I cannot confidently rely on the wall-clock comparison.

There are some additional points that I would like to phrase as questions below.

Questions for authors

As mentioned above, there are some further important points that I would like to phrase as questions to the authors.

1) Advancement over existing recency-based replay: The contribution would be clearer with stronger comparisons against simpler recency controls. Wang and Ross’s Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past already studies preferential sampling of recent experience and its combination with PER. Could you discuss this connection and compare FreshPER with a tuned recent-window baseline and age-only sampling? This would help establish the value of combining base priority with exponential age weighting beyond a general preference for recent data.

2) Age as a proxy for policy mismatch: Appendix D provides a reasonable motivation, and I appreciate that you distinguish the full-trajectory ESS relationship from the clipped optimization surrogate. However, how strongly does collection age correlate with actual policy mismatch in your experiments? Reporting policy-ratio statistics, clipping fractions, or an appropriate divergence diagnostic by age would help support the proposed explanation, particularly when different iterations contain different numbers of replay updates.

3) Interpretation of the RLEP comparison: RLEP retrieves only 12 historical trajectories across six of its 75 main-training groups. This means its replay mechanism is barely exercised in this setting. Could you clarify whether the result primarily demonstrates limited success-pool coverage under the chosen budget, rather than the relative quality of its replay strategy? A coverage analysis or an additional configuration where matching successes are sufficiently available would make this comparison more informative.

I would appreciate it if you could address these points!

2 comments

Thank you for your detailed review and for recognizing our practical implementation (behavior log-probs, group GRPO replay, PPO value/GAE refresh) and transparent reporting. We address each point below:

  • W1 (Isolating freshness on larger tasks): Our logs already contain the exact single-factor ablation on ScienceWorld (Qwen3-8B, GRPO) keeping code, buffer (64 8-trajectory groups), base priority, α=0.6\alpha=0.6, IS β\beta schedule (0.4→1.00.4\to 1.0), LR, and budgets identical while toggling only enable_age_decay. Both FreshPER runs beat both Priority runs (18.54/21.88 vs. 18.39/17.63, mean 20.21 vs. 18.01). Across all 25 strictly matched pairs in our logs, FreshPER wins/ties in 21 cases (18 wins, 3 ties, 4 losses). We will separate this matched matrix from the tuned curve in Fig. 4(b) and label NQ Search as a full-configuration comparison.
  • W2 (Fig. 4(a) vs. 5(b)): The two plots come from different run batches: Fig. 4(a) shows our main runs, whereas Fig. 5(b) came from an earlier profiling batch whose On-policy run collapsed early. We will remove this cross-batch curve from the main text and report same-batch per-iteration compute breakdowns (rollout vs. replay step time) and checkpoint logs in the appendix.
  • Q1 (ERE & recency baselines): Unlike ERE (Wang & Ross, 2019), which samples uniformly from a shrinking hard window, FreshPER combines base priority with smooth decay (p‾i=pibaseexp⁡(−Δi/τ)\overline{p}_i = p_i^{\text{base}}\exp(-\Delta_i/\tau)), keeping rare high-reward trajectories reachable while penalizing staleness. In Eq. (1), τ→∞\tau \to \infty recovers Priority and α=0\alpha=0 gives uniform sampling; our τ∈{500,1000,1500}\tau \in \{500, 1000, 1500\} ablations (Fig. 6) confirm timescale sensitivity. We will add this discussion to Section 2.
  • Q2 (Age vs. policy mismatch): Controlling for outer training iteration across all 22 replay runs, batch age correlates positively with out-of-clip token fraction and ∣approx. KL∣\vert{}\text{approx. KL}\vert{} in every run (partial Spearman ρ=+0.51 to +0.65\rho = +0.51\text{ to }+0.65 on ScienceWorld; +0.13 to +0.52+0.13\text{ to }+0.52 on Sokoban), validating age as a local mismatch proxy.
  • Q3 (RLEP coverage): Agreed. Under our 3,200-trajectory budget, RLEP matched only 6/75 groups (12 trajectories), reflecting limited pool coverage in multi-variation tasks rather than a flaw in RLEP's rule. We will state this explicitly in Section 4.3.

Dear authors!

Thank you so much for your response. I would have loved to see the additional discussions in an updated PDF. Hence, I keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.