acceptodds
Withdrawn from ICLR 2027

COMPACT-Bench: One Budget Axis for Memory Compaction in LLMs and Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Methods that reduce how much of its context a language model keeps in memory are developed in four separate literatures, and each reports results in its own unit. KV-cache compression reports memory saved at fixed perplexity, prompt compression reports tokens saved at fixed accuracy, sub-quadratic architectures report recall at a fixed state size, and agent-memory systems report benchmark scores without stating how much they store. All of these methods spend the same resource, yet their numbers cannot be compared. We introduce COMPACT-Bench, which measures every method on one axis, the bytes of retained state per token of history (BPT), and asks four questions at matched budgets: how much accuracy a method keeps, whether exact or lossy retention does better, whether the system can tell what it discarded, and whether its stated confidence reflects what its memory still supports. Every self-report is checked against uncompacted controls in which the fact is present or absent. On 21 settings of 16 open models from 0.5B to 24B parameters we find four results. First, the literatures disagree once every stored byte is counted. On Qwen3-1.7B a 2-bit KV cache keeps 0.98 accuracy at 22% of the full cache, where eviction, prompt compression, and summarization keep at most 0.56 within 30%, while on Qwen2.5-1.5B the same quantizer fails below 8 bits. Second, tolerance to eviction depends on model generation and not on size. In five same-family pairs from 0.5B to 14B, and across all 21 settings, every model released from November 2024 onwards loses half its accuracy at 33–69% of its budget, against 76–86% for every earlier model. Third, content-based scorers fall below random eviction only because they cannot see the question, and letting them see it recovers up to 0.98 in accuracy at the same budget. Fourth, eviction that scatters its losses leaves fragments of a fact that the model takes for the fact itself. On the models whose yes/no self-report passes both controls, SnapKV's failures are reported as still answerable in up to 74% of cases, against at most 11% for StreamingLLM, which removes facts whole, and on the models whose stated confidence falls for an absent fact, SnapKV's failures carry higher confidence than a fact that was never there. We release the benchmark and a reference implementation.

Primary area:
datasets and benchmarks
Keywords:
memory compaction, KV cache compression, long-context LLMs, prompt compression, calibration, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_yiDjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces COMPACT-Bench, a benchmark for comparing memory-compaction mechanisms using bytes of retained state per token of original history (BPT). It evaluates accuracy–budget frontiers, exact versus lossy retention, self-reported availability of information, and stated confidence after compaction. Experiments cover KV eviction, KV quantization, prompt compression, and model-generated summaries, with eviction evaluated across 21 settings of 16 open-weight models. The main findings are that mechanism rankings depend strongly on the model, query visibility substantially affects attention-based eviction, and fragmented retention is associated with models incorrectly claiming that a fact remains available. The paper also presents a rate–distortion formulation and reports an association between model release date and tolerance to eviction.

Critical strengths
  1. The benchmark examines an important failure mode beyond answer accuracy. Evaluating whether a model recognizes information loss is consequential for systems that must decide when to abstain or retrieve additional evidence. The present and absent controls help distinguish compaction-related failures from a model’s pre-existing tendency to claim knowledge. The observed contrast between contiguous removal and fragmented retention is particularly interesting, and the token-survival measurements provide a useful starting point for investigating it.

  2. The accounting framework improves the interpretability of cross-mechanism comparisons. Explicitly including quantization metadata and full-precision outliers avoids equating nominal bit width with actual retained-state cost. Evaluating mechanisms on the same questions across budget sweeps is also a sound design choice. The detailed reporting of prompts, hardware settings, harness corrections, and per-generation records supports reproducibility.

Critical weaknesses
  1. The evidence does not yet establish the breadth and distinctiveness of the proposed unified benchmark. Cross-mechanism evaluation already exists in SCBench, which also investigates shared-context reuse and degradation without query access. Similarly, LongMemEval already includes abstention. The paper’s clearest additional contribution is therefore its budget accounting combined with controlled evaluation of self-report after compaction. This contribution should be distinguished more precisely from existing work. The default query-agnostic protocol also makes the omission of KVzip, which specifically targets query-agnostic cache reuse, consequential. Without such a baseline, the results may partly reflect a mismatch between the evaluated methods and the chosen protocol. Furthermore, the cross-mechanism comparison covers only two models, while architectural memory and sustained agent-memory operation are not evaluated. Finally, BPT needs a clearer operational boundary. Summaries are charged according to their expanded KV cost, whereas soft/gist representations are listed according to their vector-storage cost. Persistent representation size, resident inference state, and peak working memory are different quantities. Their treatment must be consistent before the framework can support its broad comparability claim.
  2. Several central interpretations are not supported by the theory or experimental controls. The claim in Section 2 and Appendix A that the rate difference between query-aware and query-agnostic retention is at most H(Q)H(Q) does not hold generally. Consider a history containing nn independent random bits and a query asking for a uniformly selected bit. A query-aware encoder can retain one bit, while a query-agnostic encoder guaranteeing exact answers to every possible query must retain all nn bits. The difference is n−1n-1, which can greatly exceed H(Q)=log⁡2nH(Q)=\log_2 n. The mutual-information expression involving a single representation ZZ does not establish a bound on the optimal budgets of two different encoders. This concern is separate from the standard Fano inequality itself. The reversibility experiment also changes coverage, selection policy, and answer-time filtering alongside representation fidelity. Equal memory alone does not make the accuracy difference attributable to reversibility. Likewise, the association between retained key tokens and overclaiming does not isolate fragmentation as the cause: the probe may respond to recognition of the key without assessing whether the complete value remains recoverable. These issues affect the paper’s explanatory conclusions, even if the reported observations are reproducible.
Questions for authors
  1. Theoretical clarification: How does the claimed H(Q)H(Q) bound apply to the random-bit indexing example above? Are you bounding quantities for a fixed representation, or comparing the minimum budgets of query-aware and query-agnostic encoders?

  2. Budget clarification and baseline evidence: Which storage stage does BPT measure for each mechanism? Are the quantization points based on physically packed retained representations or an analytical storage estimate? Do the main query-agnostic conclusions hold against a method designed for that setting, such as KVzip?

  3. Experimental attribution: What evidence separates reversibility from history coverage and selection policy in Table 2? Can the conclusions be supported when the retained fact set is held fixed?

Reviewer_BfgZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author created a new metric called BPT, which is bytes per token of history, that allows a direct comparison across different compression paradigms.

Critical strengths
  1. Introduction of a unified framework for comparing across different compression methods
Critical weaknesses
  1. All evaluation tasks are synthetic. There are a lot of good long context benchmark that people usually use like RULER, Long Bench, Helmet.
  2. The context length evaluated is very short, mostly around 2k - 4k, occasionally 8k, and 1 task on 16k.
  3. SCBench already introduced a way of comparing different kv cache compression methods, including kv dropping, quantization, sparse attention, prompt compression and etc. The author did compare with SCBench, but it seems to be a little bit marginal in terms of contribution.
  4. The metric deliberately excludes compute. There might be a way to hack this in the sense that we can use a strong llm to summarize everything and obtain the best performance.
  5. Some findings such as model release date is more correlated to performance than model size are confounded. The author should use a better way to quantify those correlations.
Questions for authors

see weakness

Reviewer_BoscOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Benchmark paper. The authors point out that KV eviction, KV quantization, prompt compression, and agent‑memory summarization all spend the same resource (bytes of retained state) but report savings in incompatible units, so they define bytes per token of history (BPT), computed from the model’s L/H/d and charging quantizers for scales, zero points, and fp16 outliers. On that axis they run four tasks: a needle‑in‑filler accuracy‑vs‑budget frontier, a “reversibility” comparison of rolling summary vs raw recency buffer at equal budget, a yes/no attribution probe (does the model think it still has the fact), and stated confidence, with uncompacted present/absent controls for the last two. Across 16 open models (0.5B–24B, many in NF4) they report: mechanisms differ a lot at equal bytes and the winner flips between Qwen2.5‑1.5B and Qwen3‑1.7B; models released after Nov 2024 tolerate far more eviction than earlier ones at every size; attention‑based scorers (SnapKV, TOVA) lose to random eviction only because the question isn’t visible at compression time; and eviction that fragments a fact (SnapKV, random) makes the model claim it still has the fact on ~54% of failures, vs 3% for StreamingLLM which removes contiguous spans. There’s also a Fano‑type lower bound that the authors say doesn’t bind at any budget studied.

Critical strengths
  • The unification idea: price everything in bytes including quantizer overhead, so “4‑bit” is actually 34% of the cache, not 25%, and run every method at the same fixed budgets so nobody reports their best operating point (App. B.2)

  • The attribution finding is new and useful: pooled over six models, SnapKV failures where the probe said “yes” retained 70% of the key’s slots vs 24% where it said “no,” and random eviction shows the same pattern, so it’s fragmentation and not the scorer

Critical weaknesses
  • Single-task evaluation. All reported results derive from one synthetic needle-in-filler task (“The special magic number for {key} is {value}”) in 2,000–8,000-token contexts with 48 generations per cell, and no agent, multi-turn, or multi-hop task is evaluated despite the title and the predictions in Section 2. Because the uncompacted accuracy is 100% and the needle is lexically matched, the task is the easiest case for quantization, so the cross-mechanism ordering in Section 4.1 (2-bit retaining 0.98 vs. eviction 0.33 on Qwen3-1.7B, measured on only two ~1.5B models) is unlikely to transfer to harder workloads.

  • The release-date effect is confounded with KV-cache redundancy. The earlier group consists of five Qwen2.5 models (2 KV heads, e.g. 28,672 BPT at 1.5B) and Phi-3.5, while the Qwen3 counterparts use 8 KV heads (114,688 BPT), and the collapse point is measured as a fraction of each model’s own cache. The regression in Section 4.3 (n=16, one group label) does not include KV-head count or GQA ratio as a covariate, so the data cannot distinguish a training effect from a per-token redundancy effect.

  • Two findings reflect the evaluation protocol rather than the methods. SnapKV and TOVA are specified to score against the question in the observation window, so the 0.52 median gain when the question is made visible (Section 4.2) recovers their intended operating condition rather than revealing a new property. The reversibility task compares a summary that observes the full history with a recency buffer that discards old chunks by construction, so the measured difference is dominated by coverage rather than reversibility; a retrieval-selected raw buffer at equal budget would isolate the stated property.

Questions for authors
  • Add at least one harder task on the same axis (a RULER multi‑key/multi‑hop subset, or NoLiMa‑style needles). Does the quantization‑beats‑eviction ordering on Qwen3‑1.7B hold?

  • Can you add KV‑head count / GQA ratio as a covariate in the Sec 4.3 regression, or run one earlier model with 8 KV heads (e.g., a Llama‑3.x‑8B from 2024) and one later with 2? Right now release date and KV redundancy move together for the Qwen pairs.

  • For reversibility, what happens with a raw buffer that keeps the chunks a retriever selects rather than the most recent ones?

  • Qwen2.5‑1.5B/3B pass the present control at 0.58. Is that really “passing”? What’s the threshold?

Reviewer_FfWAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COMPACT-Bench introduces bytes per token of history (BPT) as a common budget for comparing memory compaction methods. It evaluates accuracy, reversibility, loss attribution, and confidence, revealing dependencies on the model, query availability, and how facts are retained.

Critical strengths

The paper provides a unified memory accounting framework that includes quantization overhead and extends evaluation beyond accuracy to whether models recognize information loss. Its controlled experiments offer useful observations about query conditioning and fragmented retention, supported by uncompacted present/absent controls.

Critical weaknesses
  1. The evaluation coverage does not adequately support the benchmark’s broad scope. The cross-mechanism comparison involves only Qwen2.5-1.5B and Qwen3-1.7B (§4.1), while the broader model evaluation primarily concerns KV eviction. The authors acknowledge that experiments rely on synthetic lexical retrieval, generally with 2K–8K-token contexts, without naturalistic retrieval, multi-hop tasks, or full multi-round agent-memory evaluation (§6). Consequently, the reported method rankings and reliability findings may not generalize to the diverse memory workloads motivating the benchmark.
  2. The reversibility experiment does not isolate the effect of reversibility. Section 3.2 attributes accuracy differences under matched budgets to reversibility. However, the raw buffer retains recent chunks, whereas the summary attempts to preserve information across the entire history (§4.4). Moreover, the raw-buffer arm filters retained chunks by the queried entity at answer time (Appendix B.4). These conditions also differ in coverage, content selection, and retrieval procedure. The experiment therefore compares two memory strategies without identifying reversibility’s independent contribution.
Questions for authors

The same as weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.