acceptodds
Under review as a conference paper at ICLR 2027

Remembering What Changes the Future: Learning to Compress Agent Histories for Long-Horizon Tasks

Abstract

Long-horizon agents interleave reasoning, actions, and observations, causing their interaction histories to grow until context budgets force compression. Summarization and truncation can reduce this cost, but a coherent account of the past does not reveal which facts, goals, or constraints remain necessary for future control. The central question is therefore whether a summary preserves information that can affect subsequent decisions, rather than whether it provides a fluent account of past events. We introduce Policy-Aware Compression through Counterfactual Evaluation (PACE), a reinforcement-learning framework for learning future-relevant memory. At each compaction boundary, PACE compares continuations from the same environment state: one sees the full history and the other sees a candidate compact state. The one-sided difference in downstream utility gives a direct learning signal for harmful omissions, while task-state and safety signals preserve information that may matter beyond the immediate answer. The compactor is trained using candidate groups under hard memory budgets. A hybrid-execution analysis relates local future sufficiency to repeated compaction. In 8K single-boundary evaluations on HotpotQA and MuSiQue, PACE preserves substantially more downstream QA performance than prompt-based and learned compression baselines at the same compact-memory budget. The results support a simple principle: effective agent memory should prioritize what can change the future, rather than information that merely supports a coherent account of the past.

Primary area:
reinforcement learning
Keywords:
large language model, long-horizon agents, reinforcement learning, context compaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_g65UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PACE, a reinforcement learning framework designed to train a memory compactor for language model agents. PACE optimizes the compactor by comparing an agent's future performance when given a complete history versus a compressed one from the same state. The objective incorporates three key factors: one-sided downstream regret, task-state preservation, and memory compactness, using recursive training for multi-step compression. Experiments on HotpotQA and MuSiQue with Qwen3.5-4B as both the compactor and agent demonstrate the effectivness of PACE against several compression baselines.

Critical strengths
  1. A fresh, useful perspective on learning agent memory. PACE frames compression as preserving future decision utility and makes this concrete through boundary-local comparisons between full and compressed histories. Its distinctive emphasis is on turning the harm caused by individual compression decisions into a direct training signal for the compactor.
  2. Strong results within the evaluated setting. On 8K HotpotQA, PACE achieves 81.90 F1 using approximately 135 memory tokens, compared with 73.06 F1 and 251 tokens for RECOMP-Ext. Direct transfer to MuSiQue and the rolling-compression results provide encouraging evidence that the learned compactor preserves useful information beyond its training setting.
Critical weaknesses
  1. The contribution over closely related approaches is insufficiently established. ACON already optimizes compression through downstream outcomes and full-versus-compressed trajectory comparisons, while Context Folding/FoldGRPO learns context management through RL. PACE’s boundary-local counterfactual objective is a plausible distinction, but its advantages are neither clearly motivated nor experimentally demonstrated against these approaches. Moreover, the terminal-reward ablation uses a synthetic scalar policy, and the LLM regret-weight study excludes a zero-regret condition, leaving the contribution of counterfactual feedback unclear.
  2. Limited evaluation scope. The main experiments use constructed multi-document QA histories from HotpotQA and MuSiQue, with Qwen3.5-4B as the sole backbone. One interactive evaluation, such as AppWorld or a BrowseComp search-agent setup, would more directly test these claims. LongMemEval would provide complementary evidence on long-term memory, while an additional model classes (e.g., DeepSeek-V4-Flash) would help assess model generality.
Questions for authors
  • What specific limitation of ACON’s contrastive guideline optimization or FoldGRPO’s process-reward training motivates PACE’s approach? Can a controlled comparison establish the benefit of boundary-local counterfactual feedback over one of these alternatives?

  • Could the authors provide evaluation on further benchmarks (agentic workflows or LongMemBench-like tasks) and model families?

Reviewer_48xsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies context compaction for long-horizon language agents and argues that a compact memory should preserve information according to its effect on future decisions rather than textual similarity to the original history. The proposed method, PACE, trains a compactor while keeping the downstream executor fixed. At each compaction boundary, it compares continuations from the same environment state using either the full history or a candidate compact memory. Candidates are optimized using one-sided downstream regret, compact-branch utility, task-state preservation, safety-related penalties, and an adaptive length reward under a hard token budget. Training also includes recursively compacted histories to better match repeated deployment. On padded long-context versions of HotpotQA and MuSiQue, PACE achieves substantially better QA performance than several extractive or pruning-based compression baselines. A synthetic experiment suggests that local counterfactual feedback is more sample-efficient than terminal-only RL, while a hybrid-execution analysis relates bounded local regret to performance under repeated compaction.

Critical strengths
  1. Well-motivated formulation. The paper correctly identifies that textual faithfulness is not equivalent to preserving information needed for future control. Framing compression as a future-utility-preserving state abstraction is meaningful for long-horizon agents.

  2. Appealing learning signal. Paired continuations provide localized feedback for delayed compression errors, while one-sided regret prevents improvements on some examples from canceling harmful omissions on others.

  3. Strong results under tight memory budgets. PACE substantially outperforms the evaluated compression baselines on both HotpotQA and cross-dataset MuSiQue evaluation, and the inclusion of recursive boundaries is well aligned with repeated compaction at deployment.

Critical weaknesses
  1. The experiments do not fully support the broad agent claims. The main evaluation consists of padded multi-hop QA tasks whose continuations terminate in short answers. It does not test evolving plans, tool use, irreversible actions, or long-lived safety constraints, despite these being central to the paper’s motivation.

  2. The core components are insufficiently ablated on the main benchmarks. The comparison with terminal-only feedback is limited to a hand-designed synthetic task. There is no HotpotQA / MuSiQue ablation of counterfactual regret, recursive training, task-state supervision, or adaptive memory. In particular, the regret-weight experiment does not include a zero-regret condition.

  3. The baseline comparison does not fully isolate the source of the gains. The evaluated baselines are primarily extractive selection or pruning methods. A same backbone prompted summarizer, a same model terminal reward baseline, or a supervised supporting-fact compressor would better determine whether the gains arise from the PACE objective rather than question-conditioned generation and the structured compactor instruction. The paper appropriately discusses ACON, Slipstream, and CompactionRL, but their conceptual overlap also makes a more concrete comparison or distinction desirable.

Questions for authors
  1. Can the authors evaluate PACE on an interactive agent benchmark involving multi-step tool use, evolving state, or irreversible actions?

  2. Can the authors provide main-benchmark ablations for no counterfactual regret, no recursive training, no task-state term, and no adaptive-memory mechanism?

  3. What is the performance of the base Qwen3.5-4B compactor using the same structured instruction but without PACE training?

  4. Although ACON, Slipstream, and CompactionRL are already discussed, can the authors clarify more concretely which capability PACE provides that these methods do not? If direct comparison is impractical, a controlled conceptual or component-level comparison would still be helpful.

  5. Are all methods in the 8K HotpotQA block evaluated on exactly the same task IDs? How should the 256-example main evaluation be reconciled with the 128-example RECOMP and sensitivity evaluations described in the appendix?

  6. During training, how many candidates and continuation tokens are used per boundary, and what are the total GPU hours? At deployment, what is the end-to-end latency overhead of producing the compact memory, and when is it offset by shorter subsequent contexts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.