acceptodds
Under review as a conference paper at ICLR 2027

HiDe: Hindsight Deletion of Environment-Substitutable Memory for Efficient LLM Agents

Abstract

Long-horizon LLM agents repeatedly carry historical reasoning and observations into subsequent decisions, even as environmental feedback refreshes task information and makes some records replaceable. Learning when to delete this history is difficult: task rewards provide little direct feedback on individual memory choices, while token savings alone cannot distinguish useful deletion from information loss. We introduce HiDe (Hindsight Deletion), a framework that lets the agent's realized future teach its past memory decisions through dual-level hindsight self-teaching. Its dual-view counterfactual scoring evaluates the same next realized response under the reduced history and a matched context restoring only the newly deleted blocks. With the new observation shared across both views, score differences provide an approximate measure of the response's remaining dependence on removed history. HiDe combines this dependence signal with cumulative input savings into a shared hindsight utility, discounting savings when the response remains sensitive to the deleted content. At the decision level, the utility supplies dense credit to earlier DELETE/WAIT choices; at the trajectory level, it shapes task-gated rewards to favor strong task performance and useful memory reduction. Hindsight supervision requires only forward rescoring, without additional rollouts or a separate teacher model; the trained policy makes memory decisions from its current context alone. Across WebShop, BabyAI, SciWorld, TextCraft, and SearchQA with two model scales, HiDe achieves the best or tied-best task scores among compared methods in nine of ten settings. WebShop ablations support the contribution of hindsight credit to task performance and input efficiency, while independent Keep/Delete continuations and information-support analyses examine deletion consequences and replacement evidence.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM agents, memory management, hindsight, credit assignment, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JVXJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HiDe, which trains long-horizon agents to delete history that has become substitutable by later environment feedback. At each step the agent chooses DELETE or WAIT. After deletion, the same realized next response is scored under the reduced context and under a context restoring only the newly deleted blocks; the log-probability gap estimates residual dependence on the deleted content. This dependence is combined with cumulative input savings into a hindsight utility, which supplies both decision-level credit and trajectory-level task rewards. Across the evaluated environments, HiDe generally achieves strong task-performance and input-efficiency improvements over the included baselines, although the gains are environment-dependent and the evidence is less conclusive for long-horizon deletion safety.

Critical strengths
  1. The paper introduces a coherent hindsight-training framework for learning selective history deletion from the agent’s own future behavior.
  2. The empirical study evaluates both task quality and input efficiency across several environments, providing evidence that the method can improve the score–cost trade-off in many settings.
Critical weaknesses
  1. The central supervision signal is not aligned with the claimed long-horizon deletion objective. HiDe measures the likelihood gap of one already-realized next response under deleted and restored contexts, but does not estimate whether deleting the block preserves future task success or the value of alternative future actions. A low gap can mean that the model ignores the block in this particular continuation, not that the block is safely removable. Restoring a block while holding the new observation and realized response fixed also does not establish that the environment supplied an equivalent replacement or that the information will remain unnecessary under future branches.
Questions for authors
  1. Could you evaluate an alternative signed utility in which deletions with sufficiently high dependence receive negative credit and are explicitly worse than WAIT? For example, a signed or directional likelihood gap, or an explicit harmful-deletion penalty, would test whether the current nonnegative utility induces excessive deletion.
  2. Can you quantify the training-time overhead of hindsight scoring, including the number of extra forward passes per eligible deletion, whether reduced-view rollout log-probabilities are reused, wall-clock time, GPU hours, peak memory, and throughput? A break-even analysis would clarify how many deployment episodes are needed for the reported input-token savings to amortize this additional training computation.
Reviewer_6o7kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HiDe targets to the memory deletion problem for long-horizon LLM agents. a framework that lets the agent’s realized future teach its past memory decisions through dual-level hindsight self-teaching. HiDe is a framework that lets the agent’s realized future teach its past memory decisions through dual-level hindsight self-teaching. Its dual-view counterfactual scoring evaluates the same next realized response under the reduced history and a matched context restoring only the newly deleted blocks. It post-trained the LLM with signal of dual-view counterfactual scoring. The optimization itself is a standard GRPO trajectory objective with an auxiliary PPO term on the decision span, and step-level group-relative advantages have precedent in agent RL.

Critical strengths

1, the credit signal is cheap to get given the dual-view scoring needs no extra rollouts, no separate teacher at TL time or no constructed target.

Critical weaknesses

1, the optimization itself is a standard GRPO trajectory objective with an auxiliary PPO term on the decision span, and step-level group-relative advantages have precedent in agent RL, the novelty mainly relies on the signal mentioned above. 2, for eq. 4 and 5, the decision-level loss can never reinforce WAIT and can't rank harmful DELETE below WAIT

Questions for authors

1, clarify the scenarios for eq. 4 and 5 how it reinforces WAIT and rank harmful DELETE below WAIT. 2, In "no g_t", is w(g_t) set to 1 or is the bonus removed?

Reviewer_wQVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiDe trains an LLM agent to delete historical reasoning and observation blocks from its context. After a deletion, it scores the next realized reasoning-and-action response under the reduced context and a matched context that restores the deleted blocks. The average absolute log-probability gap discounts the cumulative input savings attributed to that deletion. The paper evaluates Qwen3-4B and Qwen3-8B across five environments, reporting the best or tied-best task score.

Critical strengths
  • Using the next realized response as the supervision target. Restoring the deleted blocks while fixing the new observation and response prefixes gives a well-defined measure of response sensitivity. The local feedback requires forward rescoring, and it preserves the actual invocation context for policy updates.

  • The paired deployment experiment is the strong result. On the same 200 WebShop tasks, enabling deletion reduces cumulative input tokens. The independent Keep/Delete experiment is also useful: it evaluates downstream outcomes with seeds separate from the scoring probe and disables subsequent deletion effects in both branches.

Critical weaknesses
  • The ablations is confusing. In Table 3, “no hindsight” removes the local auxiliary loss but retains the gap-weighted trajectory bonus, but it does not test training without hindsight scoring. The exact reward replacement in “no (g_t)” is explicitly unconfirmed, and Appendix B.1 describes an unresolved configuration discrepancy for “no control.” A comparison against the same deletion policy trained with task reward and raw token savings, without the dependence signal in either objective, is needed to establish.

  • The validation of the hindsight gap is coarse to establish how reliably it identifies harmful deletions. The probe examines only the next response, although its utility credits savings over all remaining invocations. The independent Keep/Delete continuations are a useful test, but Table 2 reports only three bin averages, without task-clustered uncertainty or the fraction of harmful deletions within each bin.

  • The main comparison is not documented well. Appendix explicitly leaves checkpoints, splits, sample counts, baseline configurations, and evaluation seeds unspecified for environments other than WebShop. The SearchQA dataset mixture is also not presented.

Questions for authors
  • Among the existing Keep/Delete continuations, how often did a low-gap deletion reduce the final task score by more than 0.05? Could you show one such case and identify what information was deleted and when it became necessary?

  • How much training overhead does dual-view rescoring add in the WebShop 8B run? Are the deleted-context log-probabilities reused from the rollout, and what fraction of training time is spent on the additional scoring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.