- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FreshPER, which combines replay priorities with exponential decay in trajectory age for LLM reinforcement learning. It retains behavior-policy probabilities and integrates replay with PPO, GRPO, and a ROLL REINFORCE++ variant. Experiments on Sokoban, search, ScienceWorld, and visual tasks suggest that favoring recent experience can improve learning from a fixed rollout budget.
Critical strengths
-
The paper addresses a practical bottleneck: expensive environment interactions make trajectory reuse valuable, but useful past experience can become stale as the policy changes. FreshPER is a simple intervention that can be added to existing optimizers. In the matched Sokoban comparison, it improves PPO success from 33.6% to 43.0% over priority replay at the same fresh-trajectory budget (Sec. 4.2).
-
The evaluation distinguishes fresh-trajectory efficiency from optimization and elapsed-time budgets (Sec. 4.4). The additional budget views make the efficiency claim more informative than a comparison based only on rollout counts.
Critical weaknesses
-
The NQ Search and ScienceWorld results do not isolate the proposed freshness mechanism. On NQ, Priority and FreshPER differ in buffer capacity, base priority, and learning-rate schedule (Table 5 and Sec. F.2). On ScienceWorld, the learning-rate and importance-weight schedules differ (Sec. F.3). Thus, the large performance gaps could arise from these choices or their interactions. The paper acknowledges this, but the matched Sokoban and supporting VLM results do not establish that freshness explains the gains in the more demanding interactive settings. This limits the main empirical contribution.
-
The stability of the improvement is not established: each curve represents one training run (Sec. 4.1), so repeated evaluations along a curve do not quantify training-seed variability. This matters particularly for the strongest controlled result. With alpha = 0.6, tau = 500, and a 250-iteration Sokoban horizon, age weighting alone changes the relative sampling weight of equally prioritized trajectories by at most about exp(0.3) = 1.35. A modest sampling tilt can plausibly have a large cumulative effect, but independent repeats are needed to distinguish that effect from training variability. Given the small methodological change, reliable empirical gains carry much of the case for acceptance.
Questions for authors
Please check weaknesses.
Reply for Reviewer jif9
Thank you for recognizing our practical motivation and multi-budget evaluation framework. We address your two main concerns below:
enable_age_decay. Both FreshPER runs outperform both Priority runs (18.54/21.88 vs. 18.39/17.63, mean 20.21 vs. 18.01). We will separate this matched matrix from the tuned curves in Section 4.3 and label NQ Search explicitly as a full-configuration comparison.Furthermore, across all 25 strictly matched pairs in our logs (Sokoban REINFORCE++/GRPO/PPO and ScienceWorld GRPO), FreshPER wins/ties in 21 cases (18/3/4). We will include the full per-run table in the appendix and note in Limitations that these repeats reflect execution-level variance under a shared seed rather than independent random seeds.