acceptodds
Withdrawn from ICLR 2027

Active Forgetting for Multi-Turn LLM Agent

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-turn LLM agents rely on interaction histories to integrate evolving instructions, observations, and actions into subsequent decisions. However, retaining the full history can reinforce outdated assumptions in earlier model responses, while indiscriminate deletion can remove information essential for reasoning and tool execution. Existing approaches to history removal, compression, and memory management generally lack an explicit account of these dependencies, leaving unresolved what agents should retain and what they can safely forget. We introduce *ACTive Forgetting (ACTF)*, a training-free context selection mechanism that determines retention through explicit dependencies among interaction records. ACTF removes unreferenced assistant responses and superseded state while preserving current evidence and its required prerequisites, yielding a compact context satisfying the declared retention and dependency constraints. On the frozen Lost-in-Conversation suite, merged ACTF improves accuracy from 44.2% to 61.0%. A separate rendering audit reaches 61.7% while using 18% of full-history input tokens; controlled evidence-delivery and final-answer evaluations retain the gain. Complementary tool-use and state-update experiments show why selective retention matters: forgetting unsupported model outputs can improve reasoning, whereas preserving required interaction records avoids the failures caused by indiscriminate deletion. These findings highlight active forgetting as a promising approach to building more reliable and efficient LLM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-turn LLM Agent, Context Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_U7PvOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is concerned with using "active forgetting" . The basic goal is to use more "explicit" mechanisms to determine what systems can forget.

Critical strengths

I the the paper makes a fine argument as to what is missing from existing approaches to "forgetting" and so does a good enough job of pointing out the research gap that it will address.

Actual technical proposal seems straightforward enough, and could likely be reproduced.

Experiments seem reasonably convincing in a quantitative sense, though it's hard to tell the extent to which the numbers I'm looking at actually represent the state-of-the-art for this task.

Critical weaknesses

I couldn't really understand the contribution from the abstract and intro (or the figure). I sort of understand what the problem is (see strengths) but I don't follow at all what is happening in the examples, or what the intellectual contribution will be toward solving it. If this isn't understandable from the abstract + intro, it'll limit the readership of the paper.

Hard to tell if results are SoTA. They seem more like "reasonable baselines" but it is hard to get a sense of the gap between this and high-performing bespoke systems for these same benchmarks. Perhaps this is ignorance on my part and could be addressed by the authors.

Technical contribution is fine though perhaps a bit incremental. I think in these days of LLM papers (which I'm not sure whether this is or not) and 50k+ submissions it can be hard to get excited about ideas that seem straightforward unless the results are very clear and the intellectual contribution more compelling.

Questions for authors

See weaknesses above.

Reviewer_VfjnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The question being asked is relevant, but the theory is trivial and the results are not comprehensive or compelling.

Critical strengths

THe question bwing asked is indeed relevant as to what should be kept as relevant, paired statistics are relevant and model coverage is broad including both closed and open source models.

Critical weaknesses

the main LiC gain is "delete all assistant messages", which has already been studied before (Huag et al, 2026). Additionally, on GPT-4o-mini, plain "remove every turn" beats ACTF: 0.678 vs 0.656 (Table 17, p.23).

Dependency closure and state keys, which are the new secitons, show no measured benefit on any real benchmarks. The only comaprison that make ACTF look good is weak. Keeping a tool result together with its call is already required by the OpenAI and Anthropic APIs, which the paper itself cites. Additionally, state replacement is never tested on a real task, only on constructed cases supplied by the authors.

LiC can accept an answer before all evidence arrives, so multi-turn policies get repeated chances to answer, so this is helped by evaluation details and not forgetting.

Proposition 1 (minimality) says the closure of a set under a relation is the smallest closed superset of it, which is a standard fact. Propositions 2 and 3 are also immediate. THus, the theoretical contribution is trivial.

Questions for authors
  1. On LiC, ACTF and "remove every turn" are identical selections (A.2). What does the paper add beyond the assistant-omission results of Huang et al. (2026)?
  2. Is there any real benchmark, with metadata the environment actually provides, where dependency closure or state replacement improves task success or cost over (a) full history, (b) results-only, or (c) observation masking?
  3. Under one final answer with standard simulator delivery, the gain is +0.039 (p > 0.18). Which LiC number should readers take as the main effect, and why?
Reviewer_VgysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ACTive Forgetting (ACTF), a training-free context selection mechanism for multi-turn LLM agents. ACTF constructs an active basis Bt\mathcal{B}_t by retaining current inputs and latest state observations while omitting unreferenced assistant outputs, and then recursively restores required prerequisites through protocol-level dependency closure Kt\mathcal{K}_t (e.g., tool call-result links and explicit references). Evaluations are conducted on Lost-in-Conversation (LiC), MultiChallenge, LongBench HotpotQA, BFCL, and synthetic mechanism suites.

Critical strengths
  1. Clear Motivation and Intuitive Design: The paper addresses an important tension in agent memory management: retaining outdated information can interfere with reasoning, while indiscriminate deletion may disrupt task dependencies. The proposed framework is simple, intuitive, and training-free.

  2. Comprehensive Experimental Analysis: The paper provides extensive evaluations, insightful diagnostic experiments, and rigorous statistical analyses. The transparent reporting of negative results and experimental limitations is also appreciated.

Critical weaknesses
  1. Despite claims of dependency-aware selection, the core dependency closure mechanism is never triggered on the primary benchmarks (LiC, MultiChallenge, LongBench) due to the absence of explicit protocol metadata. Consequently, the approach degenerates into a simple heuristic of removing all historical assistant responses, offering limited novelty over existing omission baselines.

  2. In environments where the dependency mechanisms are actually active (e.g., BFCL), the approach yields no significant accuracy improvements and fails to reduce token consumption.

  3. The framework heavily relies on unrealistic "oracle" metadata (explicit state keys and reference pointers) supplied by the environment. Without these manual annotations, the method cannot handle implicit natural language dependencies, leading to notable performance regressions in revision tasks like "Version editing".

Questions for authors
  1. Given that ACTF degenerates into a simple assistant omission heuristic in the primary benchmarks, how do you justify attributing the performance gains to the dependency closure mechanism?

  2. Can you provide a realistic evaluation where the full ACTF mechanism (including state keys and reference closures) simultaneously improves task accuracy and token efficiency without relying on manual metadata engineering?

  3. How can ACTF be adapted to handle incremental delta updates and implicit language references where deleting intermediate text currently causes computational failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.