- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines a discovery–consolidation gap in gradient-free test-time reinforcement learning: a frozen LLM agent may discover high-return behavior but fail to sustain it across later attempts at the same task. It analyzes how JitRL's repeated updates relative to a frozen reference policy can reverse earlier action preferences, including a counterexample showing declining expected return despite exact advantage estimates. To address this, HIRE stores the highest-return action prefix and compacted task information separately from policy adaptation, probabilistically replays the prefix, and then continues exploration with JitRL; other attempts start afresh. The aim is to build on previously achieved progress rather than repeatedly rediscover it. The paper formalizes consolidation through expected shortfall below the best verified return and proves a return floor under faithful replay and nonnegative continuation. Experiments on ScienceWorld and WebChoreArena report gains over the evaluated baselines, with backbone comparisons and component ablations examining their scope and sources.
Critical strengths
- The fixed-reference counterexample shows that performance can fall even with exact advantages. This identifies a specific failure of the update rule and gives a concrete motivation for preserving successful behavior separately from advantage estimates.
- The reported gains span graded science tasks and sparse-reward web tasks. Improvements across several backbones suggest that replay could be useful for a range of frozen agents, although those backbone comparisons cover only two ScienceWorld topics.
- Disabling replay hurts performance even when compacted information and advantage correction remain enabled. This supports a distinction between remembering task information and executing a saved sequence. The other component ablations also help explain the contribution of the combined system.
Critical weaknesses
- The formal retention rule cannot account for the replay in Figure 6. It saves a prefix only after a strict return improvement and chooses the first step attaining the maximum. Under the stated binary terminal rewards, failed attempts cannot update an empty record initialized at zero; a negative lower bound would retain only the first zero-return action. Figure 6 nevertheless replays seven actions from unsuccessful attempts. An additional reward or storage rule is needed to reconcile the algorithm with this example.
- It is unclear what HIRE adds beyond established return-then-explore methods and successful-trajectory caching. Go-Explore [1] already separates remembering, returning, and exploring, and Intelligent Go-Explore [2] applies this framework to foundation-model agents. Neither is discussed. HIRE differs through its single best-return prefix and executable replay, whereas IGE assumes simulator-state restoration. The existing no-advantage ablation is useful, but it does not isolate partial-progress recovery from replaying complete successful solutions. Average success counts both, so the gains alone do not resolve this distinction.
- The consolidation guarantee assumes reliable recovery rather than establishing it. Faithful reward reproduction and nonnegative continuation ensure the recovery branch preserves the retained return; the proof then bounds its mixture with fresh exploration. Figure 5 resumes new actions at return 67 after an earlier return of 75, and Figure 6 uses unspecified interface repair. The conditional proof is sound, but the actual recovery procedure and its failure rate remain unclear.
- The reliability of the reported gains is difficult to assess. Budgets and reference hyperparameters are provided, but independent-run counts, uncertainty estimates, and exact task/reset identifiers are not. Successive adaptive rollouts are not independent replications. Baseline adaptation configurations and total inference costs also need enough detail to check whether the comparisons are matched.
[1] Ecoffet et al. First return, then explore. Nature, 2021. Paper.
[2] Lu, Hu, and Clune. Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models. ICLR, 2025. Paper.
Questions for authors
- What rule retains prefixes before the first WebChoreArena success? Please provide pseudocode explaining Figure 6 and distinguish any auxiliary selection signal from the return used in the theorem.
- How often does recovery reproduce the retained best return, and what shortfall remains after continuation? What determines early handover and when interface repair is used?
- On the existing ablation subset, how does HIRE compare with JitRL plus a cache of complete successful trajectories, matching compacted information, replay probability, and action/model budgets? Separating time to first success from post-success returns would help identify the benefit of partial-progress recovery.
- How many independent adaptation runs produced the main results? Please report task-level uncertainty and specify the task/reset protocol and baseline inference accounting.