acceptodds
Desk rejected at ICLR 2027

Prompt-Level Adaptive Memory Compression for Detecting Unreliable Memory

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents rely on long contexts held in working memory, from requirements retained by coding agents to evidence gathered by research agents. Compressing this memory lowers reading cost, but lost or altered details can make it unreliable for a given prompt, letting errors propagate through later actions. Existing compression methods treat compressed memory as a fixed input and offer little way to tell when it should not be trusted. We formulate prompt-level adaptive compression, in which an agent uses compressed memory while identifying, prompt by prompt, when it is unreliable, and propose Gated Context Memory (GCM). GCM stores context as continuous latent states produced by a writer and consumed by a reader that answers the prompt: a simple writer retains prompt-relevant information without converting it into text, and the confidence of a reader trained to use this memory can indicate when the memory falls short of what the prompt requires. The writer and reader share one frozen LLM with a small set of trained parameters. The trained reader supplies this signal at its first decoding step, without generating a complete answer, consulting an external verifier, or training a separate failure predictor, and the signal can drive any correction strategy. Across tool use, multi-hop reasoning, and long-document question answering, GCM outperforms existing compression baselines, and with raw-context fallback it matches or exceeds raw-context performance for most readers. Transfer experiments across tasks, readers, and compressors show where the signal generalizes and where it breaks down.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory Compression, Robustness, Adaptive Compression, Prompt-level Adaptive.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Sak1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Gated Context Memory (GCM) for detecting when compressed context lowers answer quality relative to raw context. A prompt-conditioned writer extracts intermediate LLM states into latent memory, and a LoRA-adapted reader learns from task, distillation, and reconstruction losses. The reader’s first-token confidence determines whether to continue with memory or fall back to raw context. Experiments cover BFCL, HotpotQA, NarrativeQA, and LongBench transfer, with additional signal comparisons and ablations. Gating often recovers raw-context performance, though detection quality and fallback rates vary substantially across settings.

Critical strengths
  1. First-token confidence is a useful and inexpensive signal once memory has been constructed. This provides a concrete example where the signal supports selective use of compressed memory.

  2. The ablations and negative results help establish the method’s limits. In particular, near-chance detection on NarrativeQA and Qwen3.5-9B LB-2WikiMQA makes clear that improved task scores do not necessarily imply successful failure detection.

Critical weaknesses
  1. The comparisons do not isolate the benefit of compression. LongBench gives GCM 128S states but textual baselines only 128 tokens. GCM also has an adapted reader, while the raw branch does not, and the first-line scorer can include reasoning markers. Budget, task adaptation, and output format could therefore explain part of the gains.

  2. The practical efficiency benefit remains unclear. Table 15 reports higher encode-plus-read latency than raw reading in all four profiles, before fallback. Memory is rebuilt for each prompt, and HotpotQA fallback reaches 60–80% in Table 3 and over 90% in Table 8. The reader savings do not yet establish a lower-cost system.

Questions for authors

Do the gains and detection AUROC persist when scoring the final answer?

Reviewer_PSvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve the reliability of compressed context for LLM agents. By identifying that existing methods treat compressed memory as a query-agnostic output, the authors propose prompt-conditioned adaptive compression and design GCM (Gated Context Memory), which includes prompt-based LLM writer and reader that process the compressed context in hidden states. In practice, the same LLM backbone is reused for both reader and writer, with the additional trainable parameters being writer-side latent prompts and MLP, and reader-side LoRA adapter. During inference, the writer processes the context in chunks and encodes them into hidden states, the reader LLM reads those states and the prompt, and its first output distribution decides whether to keep decoding based on that compressed memory states or fallback to raw text. The authors train and evaluate GCM on BFCL, HotpotQA and NarrativeQA datasets from different splits and test the transferability to LongBench, across 5 different model backbones. Despite the results show that GCM outperforms the compared baselines in most settings, the context capping, and high fallback ratio can make them less convincing and worth further explorations.

Critical strengths
  1. Comprehensive experimentation and analysis. The authors provide a wide range of experiments beyond the main experiment results, including but not limited to zero-shot transfer to LongBench, alternative writer experiment, gating signal comparisons, and loss function ablations. These support the GCM's design choice and make its effective scope clear.
  2. Efficient method design. The only trainable components in GCM are the latent prompts, a two-layer MLP projection for the writer and a rank-64 LoRA adapter for the reader. Therefore, the cost of adopting the approach is mild, and modular design also makes it compatible with other reader/write choices.
Critical weaknesses
  1. Fallback ratio is high, so the proposed the compressed memory in underutilized. In Table 8, the fallback is 95.0% on BFCL and 91.8% on HotpotQA; in Table 3, the numbers ranges from 60.4% to 80.2% across models. These high fallback ratios suggest that the models frequently use the raw text, rather than the compressed memory for the answer. Without such gating mechanism, GCM losses most of its advantage in some scenarios such as in Table 1, therefore challenging the compression design.
  2. Evaluation settings are not entirely sound. (1) The raw-context cap is not required by all the baselines and keeps every task’s input within the model’s context window, so compression becomes unnecessary. Compression should be mainly used and evaluated when the input goes beyond the model's context window. (2) In Table 2, some LongBench raw-context scores fall below the no-context baseline, and the raw-context performance in Table 2 also deviates too much from that in Table 1, which might suggest a problematic configuration or confounding factor such as output limit.
Questions for authors
  1. Why does the raw-context baseline perform well in Table 1 but considerably worse in Table 2?
  2. Could the authors report the performance of the baselines included in Table 1 on LongBench (Table 2) as well?
  3. How is the raw-context baseline prompted and scored?
Reviewer_qKF2OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors define "prompt-level adaptive compression" which lets the LLM answer from compressed memory and detects if the answer is worse with memory compared to the LLM answering with the raw context instead. For their method they squeeze a long context into 128 vectors, which are basically extra placeholder vectors. They then run the model for half the layers which fills the placeholder values via attention and then the full model gets the question combined with these memory vectors. They evaluate on BFCL, HotpotQA and NarrativeQA.

Critical strengths

Its a good framing of the problem and defining reliability relative to raw context is clean.

Critical weaknesses

It seems that the gated results come from basically falling back to the raw context. In Table 8 the fallback ratios are reported and it falls back over 90% of the time. Also the GCM reader has a task-trained LoRA which the raw-context baseline has not, so one would already expect performance differences based on that so the natural baseline would be raw context + LoRA which I couldnt find in the manuscript.

The intro says that GCM is the strongest compression-only method on all three tasks which seems not true when looking at the results.

Questions for authors

With the above mentioned LoRA on the raw context, do the gains from GCM over raw reman across the benchmarks tested?

For the main tables, fallback ratios would be good to report

1 comment

Thank you for your response. The fix of the intro plus the fallback ratios and random control are helpful. My main concern is still the missing raw context + LoRA baseline as your SQuAD v2 SFT reference (93.1 vs. about 64.8 for gated GCM) suggests that somehow task adaptation can explain much of the gain?

Since most of my points were addressed, I’m raising my score to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.