acceptodds
Under review as a conference paper at ICLR 2027

Learning To Compress Context Into Reusable Experience Memory

Abstract

Historical examples and execution experience provide useful context for language models, but repeatedly processing their full text increases input length, prefill computation, and key/value (KV) cache requirements. We propose a context compression framework that converts retrieved experience into reusable experience memory for a frozen language model. The framework first induces common rules from training examples, then jointly encodes each rule and its associated example into a small set of layer-wise KV states. Segmented attention routes information through these memory positions, while compressor-side LoRA, task supervision, and distillation from a full-context view train the compressor to retain information relevant to answer or action prediction. At inference time, the frozen model reads the selected memory alongside the current input, separating experience encoding from online use. Experiments with Qwen2.5-7B-Instruct and LLaMA-3-8B-Instruct on ALFWorld and ContractNLI show improvements over textual retrieval baselines and competitive performance relative to parameter-efficient fine-tuning. In the ContractNLI efficiency evaluation, with caches precomputed and resident on the GPU, 15 memory positions replace approximately 1,347 experience tokens on average, yielding a \(4.83\times\) online speedup over processing the original context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Learning, experience memory, context compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rQ6cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the problem of increasingly long contexts when using LLMs as agents. However, I find the method very unclear, so I can only provide my tentative understanding. The proposed approach first abstracts rules from examples and then trains an LLM with LoRA to learn how to retain information.

Critical strengths
  1. The paper provides a comprehensive overview and categorization of prior methods.

  2. The experiments on ALFWorld and ContractNLI are consistent with the application scenarios claimed by the authors.

Critical weaknesses
  1. The introduction lacks coherence. First, the discussion of prior methods in the third paragraph does not establish a clear logical connection to the authors’ motivation and proposed idea in the fourth paragraph. Second, the fourth paragraph describes the authors’ approach with “we summarize …,” but its relationship to the subsequent introduction of latent context compression is unclear. As a reader, I find this progression confusing.

  2. The central motivation appears to offer limited novelty. The authors state that “useful information includes both specific conditions and operational details, as well as decision patterns shared across examples.” However, this categorization appears to be based primarily on intuition. The proposed method then adds a corresponding classification step, whose novelty is not sufficiently established.

  3. The method is insufficiently clear. Please see the specific questions below.

Questions for authors
  1. What is the rationale for placing the related work section after the method section and before the experiments?

  2. The paper states: “Let q denote the current input and a = (a1, . . . , aNa ) the target output of Na tokens, including the end token.” What exactly does aa represent? The sentence also appears grammatically incomplete.

  3. Section 2.1 states: “A retriever selects K examples from an experience bank based on q.” How is this implemented? For example, does the retriever use a neural model or BM25? Does the selection involve retrieval, reranking, or both?

  4. The paper states: “Given a pretrained language model pθ, we compress the selected examples and their associated rules into memory Mq with at most B positions.” How exactly is this compression performed? Is it done manually or using an LLM? If an LLM is used, what is the prompt?

  5. What does c1c_1 represent in Figure 1(b)? It does not appear to be defined in the main text.

  6. Could the authors include additional baselines, such as LLMLingua-2 and Latent Context Compilation?

Reviewer_X4D4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compresses retrieved experience into a small set of layer-wise KV states for a frozen LLM. It first induces rules from training examples, then jointly encodes each rule and example using a shared compressor. The compressor is trained with task supervision and distillation from a full-context view. Results on ALFWorld and ContractNLI show gains over textual retrieval and competitive performance with LoRA SFT. The paper also reports reduced cache storage and faster inference with precomputed memory.

Critical strengths
  • On the evaluated benchmarks, the method shows meaningful improvements compared to the case where no parameter update is allowed.
  • The method is well motivated. The ablation study explains the role of memory size, retrieval depth and training losses.
Critical weaknesses
  • The paper does not compare with the closest learned compression methods. As Section 3.1 notes, 500xCompressor already uses encoder-side LoRA, layer-wise KV memory, and a frozen reader. The current baselines do not show what the proposed approach adds under the same training data and memory budget.

  • The benefit of adding rules is mixed. Although rules help on ContractNLI, they bring only a small overall gain for Qwen on ALFWorld and slightly hurt LLaMA. More noticeably, Cool success drops for both Qwen and Llama. Since rule induction is a main contribution, I would like a clearer analysis of when these rules help and when they cause errors.

Questions for authors
  • How do different numbers of examples compare with the total memory fixed at 15 positions?
  • How well does rules-only compression work with the same memory budget?
  • Why does adding rules hurt the Cool tasks on both models?
Reviewer_hNwEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for compressing retrieved experience such as historical examples and, for interactive tasks, execution trajectories into a small, reusable KV-cache experience memory that a frozen reader LLM can condition on instead of the original text. For each group of training examples, an induction model first summarizes shared applicability conditions/procedures into a rule, each rule is then paired with one example to form a source unit. A shared compressor appends a small learnable matrix Z to each source unit and encodes it with segmented attention where each unit is encoded causally within its own block, and only its designated memory positions carry information forward producing a fixed-size, layer-wise KV memory. At inference, memories for the K retrieved units are concatenated and the frozen reader attends to them to produce an answer or action, without needs to re-encoding the original source text.

Critical strengths

[+] Precomputable, reusable KV memory that's decoupled from the specific query rather than query-conditioned soft prompts is a interesting design choice, and Table 4's efficiency breakdown isolates how much of the speedup comes from cache reuse alone versus the additional benefit of shortening the representation itself. This is an interesting, "working prototype".

[+] The two-level abstraction cross-example rules + concrete examples encoded jointly is a valuable response compared to purely summarized examples which lose shared structure.

Critical weaknesses

[-] One of the key contributions of the paper is the training loss, but I don;t see how the training objective differs from a standard autoregressive loss, isn't the "new" loss (Eq. 3) only the forward pass it's computed on is.

[-] To the best of my understanding the cost accounting in Table 4 covers only steady-state, precomputed-cache inference and excludes the cost of producing the memory in the first place.

Questions for authors

Unless I missed it, I would also have liked to see a discussion on what happens if the a) wrong "KV cache" ends up being prepended, b) what happens if the total KV cache prefixed+new only exceeds the sequence legnth?

What happens to tool-calls in agentic harness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.