acceptodds
Under review as a conference paper at ICLR 2027

HIRE: Hippocampal Replay for Test-Time Agentic Reinforcement Learning

Abstract

Gradient-free test-time reinforcement learning (RL) adapts large language model (LLM) agents through experience without updating model parameters, avoiding the weight-access requirements and parameter-optimization costs of training-time RL. In this paper, we identify a discovery–consolidation gap in test-time RL: agents can discover high-return behavior yet fail to sustain the achieved returns in later rollouts. In existing advantage-based methods, each update reweights the same frozen policy using re-estimated advantages, which can reverse action preferences that previously led to success. Inspired by complementary learning systems theory, we observe that advantage-based adaptation statistically integrates past experience but lacks a complementary hippocampus-like mechanism for recovering successful behavior. We therefore propose HIRE (HIppocampal REplay), which uses replay to recover successful behavior, allowing the adapted policy to build on previously achieved progress rather than rediscover it. We prove that HIRE satisfies a return consolidation criterion, yielding an expected-return lower bound that is non-decreasing as the best verified return increases for a fixed exploration rate. Experiments on ScienceWorld and WebChoreArena show that HIRE achieves the best overall performance among the evaluated methods. Our code is available at https://anonymous.4open.science/r/HIRE-D54E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, LLM agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xmLSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines a discovery–consolidation gap in gradient-free test-time reinforcement learning: a frozen LLM agent may discover high-return behavior but fail to sustain it across later attempts at the same task. It analyzes how JitRL's repeated updates relative to a frozen reference policy can reverse earlier action preferences, including a counterexample showing declining expected return despite exact advantage estimates. To address this, HIRE stores the highest-return action prefix and compacted task information separately from policy adaptation, probabilistically replays the prefix, and then continues exploration with JitRL; other attempts start afresh. The aim is to build on previously achieved progress rather than repeatedly rediscover it. The paper formalizes consolidation through expected shortfall below the best verified return and proves a return floor under faithful replay and nonnegative continuation. Experiments on ScienceWorld and WebChoreArena report gains over the evaluated baselines, with backbone comparisons and component ablations examining their scope and sources.

Critical strengths
  1. The fixed-reference counterexample shows that performance can fall even with exact advantages. This identifies a specific failure of the update rule and gives a concrete motivation for preserving successful behavior separately from advantage estimates.
  2. The reported gains span graded science tasks and sparse-reward web tasks. Improvements across several backbones suggest that replay could be useful for a range of frozen agents, although those backbone comparisons cover only two ScienceWorld topics.
  3. Disabling replay hurts performance even when compacted information and advantage correction remain enabled. This supports a distinction between remembering task information and executing a saved sequence. The other component ablations also help explain the contribution of the combined system.
Critical weaknesses
  1. The formal retention rule cannot account for the replay in Figure 6. It saves a prefix only after a strict return improvement and chooses the first step attaining the maximum. Under the stated binary terminal rewards, failed attempts cannot update an empty record initialized at zero; a negative lower bound would retain only the first zero-return action. Figure 6 nevertheless replays seven actions from unsuccessful attempts. An additional reward or storage rule is needed to reconcile the algorithm with this example.
  2. It is unclear what HIRE adds beyond established return-then-explore methods and successful-trajectory caching. Go-Explore [1] already separates remembering, returning, and exploring, and Intelligent Go-Explore [2] applies this framework to foundation-model agents. Neither is discussed. HIRE differs through its single best-return prefix and executable replay, whereas IGE assumes simulator-state restoration. The existing no-advantage ablation is useful, but it does not isolate partial-progress recovery from replaying complete successful solutions. Average success counts both, so the gains alone do not resolve this distinction.
  3. The consolidation guarantee assumes reliable recovery rather than establishing it. Faithful reward reproduction and nonnegative continuation ensure the recovery branch preserves the retained return; the proof then bounds its mixture with fresh exploration. Figure 5 resumes new actions at return 67 after an earlier return of 75, and Figure 6 uses unspecified interface repair. The conditional proof is sound, but the actual recovery procedure and its failure rate remain unclear.
  4. The reliability of the reported gains is difficult to assess. Budgets and reference hyperparameters are provided, but independent-run counts, uncertainty estimates, and exact task/reset identifiers are not. Successive adaptive rollouts are not independent replications. Baseline adaptation configurations and total inference costs also need enough detail to check whether the comparisons are matched.

[1] Ecoffet et al. First return, then explore. Nature, 2021. Paper.

[2] Lu, Hu, and Clune. Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models. ICLR, 2025. Paper.

Questions for authors
  1. What rule retains prefixes before the first WebChoreArena success? Please provide pseudocode explaining Figure 6 and distinguish any auxiliary selection signal from the return used in the theorem.
  2. How often does recovery reproduce the retained best return, and what shortfall remains after continuation? What determines early handover and when interface repair is used?
  3. On the existing ablation subset, how does HIRE compare with JitRL plus a cache of complete successful trajectories, matching compacted information, replay probability, and action/model budgets? Separating time to first success from post-success returns would help identify the benefit of partial-progress recovery.
  4. How many independent adaptation runs produced the main results? Please report task-level uncertainty and specify the task/reset protocol and baseline inference accounting.
Reviewer_WEQHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies gradient-free test-time RL for language agents, i.e. the model adapts in-context through mutliple episodes without weight updates. It identifies a failure mode of early work (JitRL) on test-time RL, where the model is able to find good behaviors but can drift away from them due to the advantage estimation bias and variance. To handle this, the authors propose to add some hand-designed machinery including a retention mechanism that saves the best trajectory prefix of a complted trajectory, and a recovery mechanism that replays that prefix with some probability. This is motivated by complementary learning systems theory and hippocampal system. The authors compare their approach with other test-time RL methods (e.g. Reflexion) on ScienceWorld and WebChorArena, and show their method (HIRE) achieves significantly higher success rate and mean return.

Critical strengths
  • The observation on discovery-consolidation gap is sharp, and the proposed fix is simple and cheap.

  • The improvements are significant and consistent and the experiments are done with different model families.

Critical weaknesses
  • The principle of the proposed method looks somehow similar to Ecoffet et al. 2021 (which both do 'save and replay the best episode'), could the author clarify how their approach relate to it? In addition, the replay mechanism (reusing an exemplar trjectory or workflow) is relatively common in prior works.

[1] Ecoffet et al. 2021, Go-Explore: a New Approach for Hard-Exploration Problems.

  • I think being able to retain previous good trajectories implicitly favors the 'mean' metrics that the paper reports. Instead of showing mean success rate and mean return, can the author add pass@k metric, or the sample efficiency of first success?

  • A minor point is that the approach might not work across instances of the same environment.

Questions for authors
  • Since Figure 1(b) is only a single task, are there other tasks showing a similar return pattern for JitRL? It strengthens the discovery-consolidation gap if the author can consider showing multiple tasks to avoid cherry-picking.

  • Would the advantage of the proposed approach shrinks quickly when going towards highly stochastic environment instead of approximately deterministic ones, or transferring to held-out instance of the same environment/task?

Reviewer_tGmZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that gradient-free test-time RL has a "discovery-consolidation" gap, i.e. returns found early are lost later because advantages get re-estimated. HIRE adds retention (store the best-return prefix and compacted task notes), recovery (with some probability replay the stored prefix, else restart), and exploration from the handover point. Experiments are on ScienceWorld (25 tasks) and WebChoreArena (100 tasks).

Critical strengths
  1. The gains are large and consistent across tasks and backbones. For example, ScienceWorld average is 57.9 versus 29.6 for JitRL and 39.7 for EvoTest, and WCA is 56.2% versus 30.0%
  2. The ablation shows removing recovery causes the largest drop (72.0 to 58.0), so the mechanism the paper proposes does matter in that setting
Critical weaknesses
  1. The evaluation setting is very strange --- it essentially reward instance-level memorization as far as I can tell. In otherwords, each task is repeated for N rollouts, with environment reward feedback, and "return" is measured on later rollouts of the same instance. Replaying the best stored action sequence thus guarantees best-so-far return. This is exactly the statement in Theorem 1. It assumes replay reproduces the recorded rewards almost surely (fixed resets, deterministic transitions) and that the remainder has nonnegative reward. Then with probability 1−\epsilon you get at least R*. But this is a statement about replaying a log, not about consolidation of a policy. There's is no held-out task variants or generalization being tested at all.

  2. The central motivation for the method is a bit weak. While believable that advantage re-estimation reverses good preferences, this central claim rests on one illustrated graphic in Fig. 1b, and a contrived two-stage example in Appendix A. There are no diagnostics which measure preference reversals on real tasks.

  3. The experimental validation is also a bit weak. Every baseline sees/uses a different set of information. The obvious controls like Reflexion/EvoTest or plain backbone given the best stored trajectory or action list, replay + backbone without JitRL, etc. are all missing. In Tables 1-3, the results also have no seeds counts or CIs.

Questions for authors
  1. How does HIRE perform when evaluated on held-out variants of a task (changed object or product, different goal) after being exposed to the originals?
  2. Please add baselines that receive the best past trajectory and notes without JitRL's advantage correction. How many seeds underlie Tables 1-3, and what are the CIs?
  3. How often does replay fail to reproduce the prefix on WCA, and how is that handled in the evaluation return?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.