- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces COMPACT-Bench, a benchmark for comparing memory-compaction mechanisms using bytes of retained state per token of original history (BPT). It evaluates accuracy–budget frontiers, exact versus lossy retention, self-reported availability of information, and stated confidence after compaction. Experiments cover KV eviction, KV quantization, prompt compression, and model-generated summaries, with eviction evaluated across 21 settings of 16 open-weight models. The main findings are that mechanism rankings depend strongly on the model, query visibility substantially affects attention-based eviction, and fragmented retention is associated with models incorrectly claiming that a fact remains available. The paper also presents a rate–distortion formulation and reports an association between model release date and tolerance to eviction.
Critical strengths
-
The benchmark examines an important failure mode beyond answer accuracy. Evaluating whether a model recognizes information loss is consequential for systems that must decide when to abstain or retrieve additional evidence. The present and absent controls help distinguish compaction-related failures from a model’s pre-existing tendency to claim knowledge. The observed contrast between contiguous removal and fragmented retention is particularly interesting, and the token-survival measurements provide a useful starting point for investigating it.
-
The accounting framework improves the interpretability of cross-mechanism comparisons. Explicitly including quantization metadata and full-precision outliers avoids equating nominal bit width with actual retained-state cost. Evaluating mechanisms on the same questions across budget sweeps is also a sound design choice. The detailed reporting of prompts, hardware settings, harness corrections, and per-generation records supports reproducibility.
Critical weaknesses
- The evidence does not yet establish the breadth and distinctiveness of the proposed unified benchmark. Cross-mechanism evaluation already exists in SCBench, which also investigates shared-context reuse and degradation without query access. Similarly, LongMemEval already includes abstention. The paper’s clearest additional contribution is therefore its budget accounting combined with controlled evaluation of self-report after compaction. This contribution should be distinguished more precisely from existing work. The default query-agnostic protocol also makes the omission of KVzip, which specifically targets query-agnostic cache reuse, consequential. Without such a baseline, the results may partly reflect a mismatch between the evaluated methods and the chosen protocol. Furthermore, the cross-mechanism comparison covers only two models, while architectural memory and sustained agent-memory operation are not evaluated. Finally, BPT needs a clearer operational boundary. Summaries are charged according to their expanded KV cost, whereas soft/gist representations are listed according to their vector-storage cost. Persistent representation size, resident inference state, and peak working memory are different quantities. Their treatment must be consistent before the framework can support its broad comparability claim.
- Several central interpretations are not supported by the theory or experimental controls. The claim in Section 2 and Appendix A that the rate difference between query-aware and query-agnostic retention is at most does not hold generally. Consider a history containing independent random bits and a query asking for a uniformly selected bit. A query-aware encoder can retain one bit, while a query-agnostic encoder guaranteeing exact answers to every possible query must retain all bits. The difference is , which can greatly exceed . The mutual-information expression involving a single representation does not establish a bound on the optimal budgets of two different encoders. This concern is separate from the standard Fano inequality itself. The reversibility experiment also changes coverage, selection policy, and answer-time filtering alongside representation fidelity. Equal memory alone does not make the accuracy difference attributable to reversibility. Likewise, the association between retained key tokens and overclaiming does not isolate fragmentation as the cause: the probe may respond to recognition of the key without assessing whether the complete value remains recoverable. These issues affect the paper’s explanatory conclusions, even if the reported observations are reproducible.
Questions for authors
-
Theoretical clarification: How does the claimed bound apply to the random-bit indexing example above? Are you bounding quantities for a fixed representation, or comparing the minimum budgets of query-aware and query-agnostic encoders?
-
Budget clarification and baseline evidence: Which storage stage does BPT measure for each mechanism? Are the quantization points based on physically packed retained representations or an analytical storage estimate? Do the main query-agnostic conclusions hold against a method designed for that setting, such as KVzip?
-
Experimental attribution: What evidence separates reversibility from history coverage and selection policy in Table 2? Can the conclusions be supported when the retained fact set is held fixed?