acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token Pruning: Prompt Compression via Minimum Description Length and Semantic Message Anchors

Abstract

Prompt compression accelerates long-context inference by condensing extensive context into concise core messages. Existing approaches follow an extractive score-and-drop paradigm, scoring tokens or text spans via local importance heuristics and discarding low-ranked elements. Hard deletion permanently discards distributed background context, failing on tasks requiring global counts or bounds, while token pruning destroys syntactic integrity and narrative topology. We reframe prompt compression as a coding problem rather than a filtering problem. Under the Minimum Description Length (MDL) principle, a compressed prompt is a two-part code: intact evidence units answer the query directly, while compact sufficient statistics summarize omitted background context. This view yields a select-and-aggregate paradigm beyond pruning, which we instantiate as emantic essage nchors (SMA), an efficient, training-free, and state-agnostic framework requiring no reader internal states or gradient updates. SMA segments text into grammatically self-contained propositional units, greedily packs query-salient anchors while preserving relative document topology, and compiles unselected background spans into compact sufficient statistics. SMA outperforms the strongest evaluated training-free compressor by up to 9.34 points on LongBench, accelerates prefill by up to 7.55, and reduces peak KV cache by 10.9, advancing the empirical Quality-Latency Pareto frontier.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt Compression for LLMs, Minimum Description Length, Long-context Inference

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_VFQsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SMA (Semantic Message Anchors), a training-free prompt compressor that does not access the reader model's internal states. It frames compression through the Minimum Description Length (MDL) principle as a "two-part code": verbatim evidence units (anchors) plus compact statistics that summarize the omitted background. The pipeline has four stages.

Critical strengths

None

Critical weaknesses
  1. First, I think the theory part is mostly decorative. For example, Eq. 1 is not MDL in the usual sense. It is a Lagrangian relaxation of a budget-constrained objective. Prop. 3 assumes most of its conclusion. Prop. 2 is a direct application of the data-processing inequality and holds for every compressor, including R2C and EHPC.
  2. Experiment setting has a lot of problems:
  • The paper formally structure the method as prompt compression, so kv cache eviction can also be counted as baselines according to section 3.1. However, non of the SOTA KV cache eviction baselines are featured.
  • Some baseline numbers look low. For example, LongLLMLingua, which is designed for question-aware long-context compression, scores below Selective Context and LLMLingua on several categories, and only 28.49 on Qwen at 2K tokens, against 49.65 uncompressed. This is far weaker than results typically reported for it on LongBench.
  • Featured datasets are limited. The experiments are mostly conducted on LongBench.
  1. Heuristics are unjustified and there is no sensitivity analysis. Reserving 55.2% of the budget for units that match negation, hedge, or list regexes is a strong and unusual design choice. The other settings (τ_agg = 0.08, the 0.15 hedge cap, the 8-word merge threshold, and the 384-token chunk cap) are also fixed with no sensitivity study. Lemma 8 presents the derivation of 0.552 as a principled reparameterization, but the value is still a hand-chosen default.
Questions for authors
  1. Can we feature kv cache eviction baselines?
  2. Can we have more datasets for long-context? LongBench is not considered enough anymore in nowadays' agentic era, where a lot of agentic benchmarks are suitable candidates for testing the effectiveness of prompt compression methods (and if the authors are aware, prompt compression method are being used a lot in agentic models like Claude or GPT nowadays).
  3. A lot of hyper-param in the paper is fixed. Can we have ablation studies to solve weakness 3?
4 comments
AuthorsOpenReview ↗

Response to Reviewer VFQs (1/2)

We thank you for the detailed review. We answer the theory, KV-eviction, LongLLMLingua, dataset, and heuristic points below, scoping each answer to what we measured.

W1. Theory

Where to look: Section 3.2 and Appendix A.2-A.4.

MDL enters as a design principle: intact anchors play the role of a model code, and a compact residual card codes the omitted background. Equation 1 is a Lagrangian allocation objective; lambda is not estimated. The data-processing inequality (Remark 4, Appendix A.3) bounds frozen-encoder selection. Proposition 1 carries its additivity, ranking, and length assumptions, which is the scope of the bound. Remark 2 predicts the observed QASPER trade-off. The budget reparameterization is stated directly.

W2. KV-cache eviction

Where to look: Section 3.1.

Text-level prompt compression and engine-internal KV eviction act at different stages and address complementary bottlenecks. KV eviction still requires prefilling the full N-token prompt inside a self-hosted serving engine, and it cannot reduce the input tokens billed by closed APIs. SMA operates strictly at the text interface before inference, so it composes with KV eviction on self-hosted stacks; Section 3.1 states this scope.

W3. LongLLMLingua

Where to look: Section 5.3 and Table 2.

LongLLMLingua's target_token is a per-chunk ratio with thresholded token dropout, not a hard cap; the library documentation notes that realized lengths generally fall below the target and vary with the tokenizer. The compressor uses Llama-2 tokenization, while Table 1 counts reader tokens including instruction, query, and chat framing. We report the realized length and do not pad it, because padding would depart from the official implementation. At 2K the realized means are 1,761 on LongBench, 1,670 on ZeroSCROLLS, and 1,464 on LongBench v2; per-task means range from 969 (MuSiQue) to 2,240 (RepoBench-P). The lower score coincides with the unused capacity, but it is not the sole cause: Selective Context uses 2,048 tokens and scores 31.75, LLMLingua 1,945 and 29.88, and LongLLMLingua 1,761 and 28.40 on Qwen LongBench. Table 1 here comes from the corrected re-run: repairing the upstream assertion fallback in LongLLMLingua changed its Qwen 2K LongBench score from 28.49 to 28.40.

W4. Datasets

Where to look: Section 5.1 and Table 1.

LongBench v2 is now a main-table benchmark: 503 questions, about 83K average reader input, 120K window. LongBench and ZeroSCROLLS are retained. InfiniteBench and agentic benchmarks are not tested.

W5. Heuristics

Where to look: Table 4, Appendix B, and Table 3.

Table 4 varies the safeguard fraction: 0.552 to 0.75 adds 0.91 to the eight-task mean, driven by PassageCount +4.40; lowering it to 0.35 reduces PassageCount from 54.31 to 47.10. The aggregation-reserve check moves HotpotQA by -0.25 and +0.30 and PassageCount by +0.02. The 0.15 hedge cap, the eight-word merge threshold, and the 384-token encoder window are documented with their scope in Appendix B; we did not sweep them individually. As noted in Section 5.4, these rules are lightweight surface heuristics rather than an intent classifier: count and existence triggers can be noisy, while the summary and query-free branches drive the measured gains, and misrouting only alters the appended card without affecting anchor selection or the budget guard.

AuthorsOpenReview ↗

Response to Reviewer VFQs (2/2)

Q1. KV-eviction baselines

Where to look: Section 3.1.

Answered above: the mechanisms act at different stages and compose (Section 3.1). A same-table comparison cannot hold the serving stack constant, because KV eviction modifies engine-internal cache management whereas SMA operates strictly at the text interface.

Q2. More datasets

Where to look: Section 5.1.

LongBench v2 is added. Agentic traces and InfiniteBench lie outside the measured set. Agentic traces need a multi-turn controller and InfiniteBench needs a separate long-input loader; neither is part of this revision.

Q3. Hyperparameter ablations

Where to look: Table 4 and Appendix B.

Table 4 and Appendix B cover the safeguard fraction and the aggregation reserve. The three fixed constants are documented with their scope, but they are not individually swept; we state this rather than presenting a component ablation as a single-parameter sensitivity.

SMA leads EHPC by 12.94 points on Qwen and 11.04 on Llama at 2K, reaches 46.98 on Qwen 3K and 38.65 on ZeroSCROLLS, and holds the high-accuracy end of the latency-quality frontier. MDL frames the representation, and the KV comparison sits outside the text interface.

I thank the authors for the response, yet my concerns remain:

  • For (1), I'm now doubting if it's the authors write the response or if it's AI-generated, since the response seems to not answer my concern at all. Here, I state again some main points of my concern "the theory part is mostly decorative", "a Lagrangian relaxation of a budget-constrained objective", or "Prop. 3 assumes most of its conclusion". The authors reply "MDL enters as a design principle: ". If I get it correctly, are the authors saying my claim is correct that the theory part offers little theoretical contribution, and mostly serve as a design cosmetic?
  • For (2), I'm clearly asking for an experimental comparison with kv cache eviction method. The authors direct me to section 3.1, and no experiment results can be found.
  • For (4), the authors state " InfiniteBench and agentic benchmarks are not tested.". I don't quite get it why the authors need to state this in the rebuttal at all. Would it help clarify any of my questions or concerns?
  • For (5), I clearly stated at the beginning "Heuristics are unjustified", so I'm trying to ask for justification, but no justification is provided, when I go to "Where to look: Table 4, Appendix B, and Table 3.".

I stop reading after (5). I think at least the authors need to value the effort and time both the authors and at least me, the reviewer, spend on (assumingly) researching and proposing constructive feedback to improve the work further. Please take this into consideration for your next submission to respect the effort and time of, at least, reviewers who still read your full paper and write honest reviews, not AI-generated ones.

I hereby also flag this to AC, as a reviewer who has decent experience in this field, my reply to the submission's rebuttal. I therefore recommend "clear rejection".

AuthorsOpenReview ↗

Response to Reviewer VFQs

We thank you for the detailed review, and I am sorry that my previous reply pointed at sections instead of answering you directly. I did use an LLM to tighten text drafted by the authors, hoping to make all words tight and uniform so each change would be easy to locate in the revised paper. That made the response read like a checklist, but the technical positions are ours.

On the theory, I should have stated our position plainly. Equation 1 is the Lagrangian form of the budget-constrained objective, with lambda never estimated, and Proposition 1 in the revised manuscript (Proposition 3 in the submitted version) is a conditional bound. The paper claims a structural decomposition and an explicit set of conditions under which the bound holds. We do not claim a new theoretical law.

On KV-cache eviction, we have not run such a baseline, so we cannot claim an empirical comparison. Eviction rewrites engine-internal state during inference, whereas our evaluation benchmarks text-interface prompt compressors (from Selective Context and LLMLingua to RECOMP, CPC, R2C, and EHPC) under a uniform reader serving stack. We should have stated this experimental scope in Section 3.1.

Likewise, listing benchmarks we did not run, InfiniteBench and agentic traces, added nothing to the reply. We did add two trained compression baselines, RECOMP and CPC, and one benchmark, LongBench v2.

On the heuristics, we swept one parameter, the safeguard fraction tau_t1. Raising it to 0.75 adds 0.91 points on the eight-task mean, lowering it to 0.35 drops PassageCount by 7.21 points (54.31 to 47.10), and keeping 0.552 reserves room for query-ranked anchors. The 384-token encoder window caps memory during batch encoding: across 94,247 prose units from 150 LongBench documents, 7 units exceed it and the 99th percentile is 87 tokens, so it is virtually non-binding in practice. The 0.15 hedge cap and the eight-word merge threshold are engineering defaults that we did not sweep individually.

Thank you for reading the full paper and holding us to a higher standard. Even if these gaps cannot be closed in this cycle, your feedback gives us a clear roadmap for the next version.

Reviewer_q4LwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the SMA method, which aims to compress long-context prompts by combining "query-relevant semantic anchors" with "compact background statistics." The method employs similarity-based ranking to guide the selection of semantic units. Experimental results demonstrate that this approach improves model performance on the LongBench benchmark while reducing latency during the "prefill" phase and lowering KV-cache memory consumption. The evaluation covers two distinct "reader" models.

Critical strengths
  1. The paper is well-written, accessible, and features clear figures.
  2. Given the high inference costs of Large Language Models (LLMs), prompt compression serves as an effective strategy to reduce GPU memory usage and accelerate the prefill phase. The authors propose an efficient compression method capable of mitigating information loss, even at high compression ratios.
  3. The method is evaluated across various LLMs and datasets; in primary comparative experiments, it achieves a higher average score on the LongBench benchmark than other baseline compression methods.
Critical weaknesses
  1. Limited methodological innovation. SMA primarily integrates existing techniques: query-aware, sentence-level selection was previously explored in CPC [1]; summarization-based compression was discussed in RECOMP [2]; and the "rate-distortion" formulation for prompt compression was introduced by Nagle et al. [3]. Overall, the approach represents an incremental synthesis of existing concepts in context selection and compression.
  2. Lack of comparison against highly relevant baselines. Although the paper mentions CPC and RECOMP, it does not provide experimental comparisons against these specific sentence-level selection or summarization-based compression methods. Due to the lack of a comparative analysis with highly relevant methods, I cannot determine whether this framework truly offers a significant advantage over existing prompt compression workflows.
  3. The experimental section covers only the Qwen3.5-9B and Llama-3.1-8B-Instruct models; it omits other model families, larger-scale models, and MoE architectures. Since the effectiveness of "semantic anchor selection" and "background aggregation" may depend on the underlying reader architecture, additional experimental results—for instance, using models like Gemma-4-31B or Qwen3.6-35B-A3B—are necessary to better validate the framework's generalization capabilities.
  4. Evaluation is primarily based on LongBench and ZeroSCROLLS, failing to include other long-context reasoning benchmarks such as LongBench v2 [4] or InfiniteBench [5]. Consequently, the current evidence is insufficient to demonstrate whether the proposed compression strategy remains effective across a broader range of long-context tasks.
  5. Improvements on downstream tasks are not always substantial. For example, in the extended ZeroSCROLLS test, SMA achieved an average score of 20.54, compared to 21.61 for EHPC. Furthermore, when using the Qwen3.5-9B model with a 2,000-token limit, SMA’s performance on few-shot tasks is notably inferior to that of LLMLingua (52.94 versus 65.13).

[1] Liskavets B, Ushakov M, Roy S, et al. Prompt compression with context-aware sentence encoding for fast and improved llm inference [C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(23): 24595-24604.

[2] Xu F, Shi W, Choi E. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation [C]. International Conference on Learning Representations. 2024, 2024: 43478-43502.

[3] Nagle A, Girish A, Bondaschi M, et al. Fundamental limits of prompt compression: A rate-distortion framework for black-box language models [C]. Advances in Neural Information Processing Systems, 2024, 37: 94934-94970.

[4] Bai Y, Tu S, Zhang J, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks [C]. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 3639-3664.

[5] Zhang X, Chen Y, Hu S, et al. ∞ Bench: Extending long context evaluation beyond 100K tokens [C]. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 15262-15277.

Questions for authors
  1. How does SMA differ from approaches that combine existing techniques—such as sentence selection, semantic ordering, and background summarization?
  2. Could the authors provide further comparisons with related methods and alternative "reader" architectures, as well as experimental results on LongBench v2 and InfiniteBench?
  3. Could the authors explain why the performance gains are inconsistent across different tasks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer q4Lw (1/2)

We thank you for your detailed review. The revision adds the trained baselines, a new benchmark, and a direct ablation of the components, and it states the reader dependence of the results.

W1. Methodological novelty

Where to look: Section 1, Section 3.2, and Section 5.4.

The contribution is a two-part representation for a fixed prompt budget: intact semantic anchors plus a compact residual card over the omitted background. R2C selects chunks and sentences with a trained FiD reader; RECOMP trains extractive and abstractive compressors; CPC learns sentence representations. SMA is training-free and state-agnostic, and its residual budget is explicit. The ablation isolates safeguards, source order, aggregation, and plain dense retrieval, so the representation is tested rather than asserted.

W2. CPC and RECOMP

Where to look: Section 5.1 and Table 1.

Both are now in Table 1. On Qwen at 2K, SMA scores 45.85 and CPC 45.80, a statistical tie with interval [-0.78, +0.88]; at 3K, SMA scores 46.98 and CPC 46.62, and SMA ranks first. On Llama, CPC leads at 41.93 and 42.40 against 40.95 and 41.94. We report both directions rather than a one-sided win. Table 1 comes from a single re-run after three implementation conditions of the compressor and two baseline configuration defects were fixed.

W3. Model coverage

Where to look: Section 4.2.

Within our compute budget we prioritized two dense readers, Qwen3.5-9B and Llama-3.1-8B-Instruct, and a 503-question LongBench v2 suite at about 83K average reader input over additional model families. Larger and mixture-of-experts readers are untested, and the claims are scoped to the readers measured.

W4. Benchmark coverage

Where to look: Section 5.1 and Table 1.

LongBench v2 is now a main-table benchmark: 503 four-way questions, with SMA second in all four reader-budget settings (19.28 and 21.07 on Qwen; 27.83 and 27.63 on Llama; R2C first). LongBench and ZeroSCROLLS are retained. InfiniteBench is not tested.

W5. Magnitude and consistency of gains

Where to look: Section 5.1, Section 5.2, and Table 1.

Against the strongest training-free baseline, SMA leads by 12.94 points on Qwen and 11.04 on Llama at 2K, and by 11.79 and 9.11 at 3K. On Qwen 3K, SMA leads five of six LongBench categories and reaches 38.65 on ZeroSCROLLS. On Llama, SMA retains 100.2% and 102.7% of the uncompressed LongBench score; few-shot rises from 39.66 uncompressed to 50.22 with SMA, while CPC reaches 53.66. The gains are not uniform: CPC leads on Llama, and SMA trails on some single-document and code tasks. Section 5.2 reports the task trade-offs.

AuthorsOpenReview ↗

Response to Reviewer q4Lw (2/2)

Q1. Difference from combining existing techniques

Where to look: Section 5.4 and Table 3.

The difference is the explicit two-part allocation: anchors keep focal evidence verbatim, and a separate residual budget carries background statistics. A sentence selector with no residual budget sets the omitted spans to zero; a summarizer rewrites the focal evidence. The plain top-k control shares the encoder and source-order rendering but loses 18.47 on PassageCount and 3.30 on BookSumSort, which isolates the structure around the encoder.

Q2. More readers and benchmarks

Where to look: Section 5.1 and Section 4.2.

LongBench v2 is added. Larger and mixture-of-experts readers are untested. The two readers already disagree on several tasks, so the paper reports reader dependence instead of pooling.

Q3. Inconsistent task gains

Where to look: Section 5.2.

The gains concentrate where cross-span aggregation matters: PassageCount, HotpotQA, and chapter ordering. They shrink or reverse where the answer is local to one passage, such as PassageRetrieval-en, QASPER, NarrativeQA, and MuSiQue. This is consistent with the selection bound and with the aggregation criterion, and it is reported rather than averaged away.

The revision adds the baselines, LongBench v2, and the component controls. We hope this addresses the main concerns.

Thank you for the additional experiments and clarifications. I appreciate the explanation of how SMA allocates the prompt budget between verbatim evidence and compact background statistics. In my view, the differences from prior work on sentence selection and summarization remain modest, so I still consider the methodological novelty limited.

The evaluation remains limited to the original two dense readers. Experiments with larger models, additional model families, and mixture-of-experts architectures are still missing, as are results on InfiniteBench. These gaps leave my concerns about the method’s generalizability unresolved. I will therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer q4Lw (3)

Thank you for taking the time to follow up and for acknowledging the added baselines, LongBench v2 evaluation, and component ablations. We respect your assessment and would like to briefly clarify our view on the methodological distinction and the practical constraints behind the remaining scope.

On methodological distinction relative to existing baselines:

  1. We agree that SMA does not introduce a brand new neural architecture. Its core novelty is paradigmatic rather than architectural—recasting prompt compression from one-part score-and-drop filtering into a two-part MDL coding problem (select-and-aggregate), and showing how this formulation separates SMA from existing baselines:Two-part MDL paradigm vs. local information scoring and rate–distortion diagnostics: Prior information-theoretic compressors such as Selective Context and LLMLingua/LongLLMLingua use local perplexity or self-information solely as a token-pruning heuristic, effectively setting the residual code to zero and erasing global context. Meanwhile, prior rate–distortion analysis (Nagle et al., 2024) characterizes diagnostic limits of black-box pruning without prescribing prompt structure. SMA is, to our knowledge, the first framework to operationalize MDL's two-part decomposition directly within a fixed natural-language budget—pairing intact semantic anchors (the structural "model code") with a deterministic background statistics card (the "residual code").

  2. Training-free and state-agnostic vs. trained or reader-coupled baselines: RECOMP trains extractive and abstractive compressors, CPC learns a context-aware sentence encoder, R2C relies on an NQ-trained FiD cross-attention reader, LLMLingua-2 distills a token classifier, and EHPC taps internal evaluator attention heads. SMA requires neither compression-specific training nor reader internal states, yet matches CPC on Qwen3.5-9B at 2K tokens and leads at 3K.

  3. Verbatim-plus-residual preservation vs. pure selection or rewriting: Pure sentence/chunk selectors (CPC, R2C, and our plain top-kk control) discard all unselected spans, which breaks global counting and chapter ordering (e.g., plain top-kk drops 18.47 points on PassageCount and 3.30 on BookSumSort in Table 3). Abstractive summarizers rewrite focal text, while token pruners (LLMLingua-2) fragment syntax and overshoot the 2K budget on 83K-token inputs (reaching 1,919 MB KV on LongBench v2 vs. 189.8 MB for SMA). SMA's safeguarded, source-ordered propositions plus residual card under a reader-tokenizer guard close the gap to trained compressors without supervision or reader access.

On larger/MoE readers and InfiniteBench:

We fully acknowledge that testing larger models, MoE architectures (e.g., Gemma-4-31B or Qwen3.6-35B-A3B), and InfiniteBench would make the empirical validation more complete. In practice, we could not complete these runs within the two-week rebuttal period due to concrete infrastructure and compute constraints: our server environment runs legacy CentOS Linux 7.9 with eight 48 GB RTX A6000 GPUs (Appendix C), where newer MoE/30B+ inference stacks and CUDA dependencies require environment upgrades beyond what could be safely reconfigured and verified during rebuttal, and our available GPU time was fully committed to running the 503-question LongBench v2 suite (~83K average tokens) alongside CPC, RECOMP, and the bootstrap comparisons across both readers and budgets. We have documented these untested regimes as limitations and leave evaluations on larger/MoE readers and InfiniteBench to future work.

Thank you again for your constructive feedback throughout the review process.

Reviewer_kWEKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work proposes Semantic Message Anchors (SMA), a context compression method that identifies and retains important context units (sentences) defined by relevance to the specific query and summarize the remaining (via rule-based statistics such as entities counts or some other textual features).

Critical strengths

The writing is clear to follow.

Critical weaknesses
  1. Limited applicability to multi-turn settings. the introduction lists long-horizon agent planning as a motivating application, but all experiments use a single query. Because selection of context components depends on the query, content dropped for one query is lost for later ones unless the system recompresses from the raw history every turn, which cancels the benefit of reusing the cache across turns.
  2. Missing important baseline. The paper cited ACON (Kang et al., 2026), a compression method for agentic tasks, which is more relevant to the agentic settings used to motivate the work, but didn't do any comparison.
  3. Rule-based components appear tailored to LongBench. The question-type classifier, which controls the aggregation step, uses fixed keywords (App. C) that match LongBench’s query templates (e.g., PassageCount, Fig. 8). The results reflect this. The results reflect this: the synthetic category (Table 1, “Synth.”; in LongBench, PassageCount and PassageRetrieval-en) accounts for about 6.1 of the 9.34-point gain over EHPC.
Questions for authors
  1. Can the authors show how the work generalizes to agentic settings and add relevant compression methods such as ACON?
  2. How does SMA generalize to an arbitrary benchmark where question types are not easily classifiable unlike LongBench? Since the method itself used predefined keyword matching for selecting parts of the relevant context
2 comments
AuthorsOpenReview ↗

Response to Reviewer kWEK (1/2)

We thank you for the review. The revision narrows the scope to single-turn compression, adds cross-benchmark evidence for the rules, and states what the classifier is and is not.

W1. Multi-turn. SMA compresses one prompt against one query, so an agent run would recompress the raw history at each turn, the mode you describe as cancelling cache reuse. We do not claim otherwise, and Section 6 states this boundary.

W2. ACON. ACON maintains criteria across turns and needs per-turn LLM calls, so comparing it would compare two harnesses. We have not run it.

W3. Rule components. The classifier reads only the query string and never receives a benchmark identifier. Table 5 reports the branches on all three benchmarks: five of the six fire on all three, and query-free fires only when the harness passes no compressor query, which happens on LongBench alone. LongBench v2 has 24% existence and 15% extremum assignments. Your decomposition holds with a different denominator: the synthetic category contributes 6.08 of the 12.94-point margin over EHPC, against 6.1 of 9.34 in the submitted version, and PassageCount enters through the empty-query branch rather than a count keyword.

W4. Synthetic category. PassageCount uses the empty-query branch and a deduplicated residual, and the plain top-k control loses 18.47 points there. BookSumSort and GovReport take the summary or query-free branch.

AuthorsOpenReview ↗

Response to Reviewer kWEK (2/2)

Q1. Agent settings. Answered in W1: we did not do cross-turn state and did not compare to ACON.

Q2. Arbitrary benchmarks. The rules never select anchors. They decide only whether a card is appended, while similarity, the safeguard set and source order do not depend on the branch. Where no rule fires, the compressor falls back to extraction and emits the same anchors without a card, and Table 3 measures what the card is worth.

Reviewer_UVYHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper recasts prompt compression as a two-part MDL code: intact query-relevant "anchor" units plus compact statistics summarizing the discarded background. The proposed method, SMA, works in four stages. It splits context into sentence-level units with regex rules, reserves budget for "safeguard" units (negations, hedges, enumerations), and greedily packs the remaining budget by BGE-M3 query similarity while keeping document order. It then fills an 8% reserve with predicate-specific background statistics (counts, extrema, existence) and truncates to the token budget. On LongBench and ZeroSCROLLS with Qwen3.5-9B and Llama-3.1-8B, SMA beats six training-free compressors on LongBench by up to 9.34 points at a 2K budget, with 7.55× faster prefill and 10.9× lower peak KV memory. Supporting propositions cover an information ceiling, a greedy approximation bound, and conditions under which aggregation helps.

Critical strengths
  • Large, consistent LongBench gains: SMA has the best LongBench average in all four reader–budget settings, with especially large margins on multi-document QA and synthetic retrieval (e.g., 73.9 vs. ≤33.5 on Synth at 2K/Qwen).

  • Simple and practical: The method is training-free, needs no reader internals, uses an off-the-shelf encoder and enforces the budget strictly. That makes it easy to deploy and reproduce, and code is provided.

  • Good diagnosis of token-pruning failure: The qualitative analysis (Figure 2) of how token pruners fragment syntax and how chunk pruners miss evidence motivates proposition-level selection convincingly.

  • Complete efficiency reporting: Separating compression time, TTFT, end-to-end latency and peak KV memory shows the real trade-off, including that iterative LLMLingua variants are slower than no compression.

Critical weaknesses
  • Missing baselines: At its core, SMA is sentence-level dense retrieval (BGE-M3 top-k within a budget, kept in document order). It is never compared against plain sentence/chunk retrieval with the same encoder, or against extractive or abstractive compressors such as RECOMP, CompAct or CPC (Liskavets et al., cited). So it is unclear how much the MDL machinery, safeguards and aggregation add beyond standard RAG-style selection.

  • Proposition 2 restates the data-processing inequality. Proposition 3's bound has a prefactor of ~0.07 at the reported κ_l, which the authors admit is not a numerical certificate. Proposition 4 is close to tautological ("if aggregation reduces distortion more per token, it reduces distortion"). The MDL framing does not drive any design choice; λ is never estimated.

  • Background aggregation contributes little and looks fragile: Removing it costs only ~3 points. The predicate classifier is keyword regex, and the GovReport example (Figure 9) shows a garbled "[SUMMARY]" card made of truncated fragments, weakening the claimed "sufficient statistics" contribution.

  • Overstated claims on ZeroSCROLLS: The text says R2C beating SMA on ZeroSCROLLS/Qwen at 2K is the "single exception." At 3K/Qwen, however, SMA (26.16) also trails LLMLingua-2 (28.28) and EHPC (26.93). SMA also loses substantially on few-shot tasks (52.94 vs. 65.13).

  • Ad-hoc hyperparameters without variance: τ_t1 = 0.552, τ_agg = 0.08 and the 0.15 hedge cap are hand-set with no sensitivity study. All results come from a single deterministic run with no significance testing.

Questions for authors
  1. How does SMA compare with a "BGE-M3 sentence top-k in document order" baseline at the same budget, and with RECOMP/CompAct/CPC? This is the key question for judging novelty.

  2. Which LongBench subtasks drive the Synth gain (PassageRetrieval vs. PassageCount)? Is the gain from aggregation, or simply from keeping whole sentences that match a summary-style query?

  3. How sensitive are results to τ_t1, τ_agg and the regex predicate classifier? How often is κ misclassified on LongBench queries?

  4. Please correct or explain the ZeroSCROLLS claim at the 3K budget on Qwen3.5-9B.

2 comments
AuthorsOpenReview ↗

Response to Reviewer UVYH (1/2)

We thank you for the detailed review. The revision adds the baselines you asked for, corrects the ZeroSCROLLS and few-shot numbers, and adds LongBench v2.

W1. Missing baselines. The control you asked for is Table 3. Plain BGE-M3 retrieval in source order, with safeguards, aggregation and markers removed, loses 18.47 points on PassageCount, 1.75 on HotpotQA and 3.30 on BookSumSort, gains 1.60 on QASPER, and spends 19 more context tokens. The gap sits in the structure around the encoder. CPC and RECOMP are now in Table 1 beside R2C and EHPC. CompAct needs a trained iterative summarizer, which is the supervision axis RECOMP and CPC already cover.

W2. Theory. Equation 1 is the Lagrangian form of the budget constraint and lambda is never estimated. Proposition 1 states how much of the optimal subset's value the greedy pack retains, given the length ratio and the coupling term. Remark 4 records what a frozen-encoder selector can recover, and Remark 2 gives the replacement criterion the QASPER ablation is consistent with. We claim a conditional bound, not a new theoretical law.

W3. Aggregation. The effect is task-dependent. Removing aggregation costs 5.76 on PassageCount and 2.23 on BookSumSort and gains 0.80 on QASPER, and the plain top-k row shows the same pattern without any card. Figure 9 has been replaced: the GovReport example now shows the anchors as emitted, with no truncated card.

W4. ZeroSCROLLS and few-shot. These numbers supersede the submission. The earlier build violated three conditions the method requires: the unit set covers the input, each retained record is self-contained, and omission markers share the budget. Qwen LongBench moved from 42.25 to 45.85, with TREC 31.00 to 75.00, HotpotQA 55.88 to 60.04 and 2WikiMQA 52.53 to 58.34; ZeroSCROLLS moved from 20.34 to 36.71. Two baselines carried defects that we repaired: R2C scored at most 128 paragraphs, which truncated 274-paragraph documents, and LongLLMLingua returned the prompt unchanged when an upstream assertion fired. On Qwen, ZeroSCROLLS is 36.71 against RECOMP 37.03 at 2K and 38.65 against 37.74 at 3K, or 96.0% of the uncompressed score. On Llama, SMA is third at 26.75 and 28.99. Few-shot is 67.44 and 67.28 on Qwen, and 50.22 on Llama where the uncompressed model scores 39.66; CPC reaches 53.66 on Llama.

W5. Hyperparameters and significance. Table 4 varies the safeguard fraction: 0.75 adds 0.91 to the eight-task mean, 0.35 costs 7.21 on PassageCount, and the comparisons keep 0.552 to reserve budget for query-ranked anchors. The aggregation reserve moves HotpotQA by -0.25 and +0.30 and PassageCount by +0.02. The 384-token encoder window is a memory cap: 7 of 94,247 prose units exceed it, and the 99th percentile is 87 tokens. We did not sweep the 0.15 hedge cap or the eight-word merge threshold. A 2,000-replicate paired bootstrap puts SMA minus CPC at +0.05 [-0.78, +0.88] on Qwen and -0.97 [-1.96, -0.10] on Llama.

AuthorsOpenReview ↗

Response to Reviewer UVYH (2/2)

Q1. Plain top-k. SMA leads on four of the five ablation tasks and trails by 1.60 on QASPER, with 19 fewer tokens. Against trained compressors it is level with CPC at 2K on Qwen (45.85 against 45.80) and ahead at 3K (46.98 against 46.62), and leads RECOMP by 2.59 [+1.71, +3.53]. CPC leads on Llama by 0.98 and 0.46.

Q2. Subtasks. The synthetic margin comes from PassageCount (6.76 against CPC) rather than PassageRetrieval-en (-3.50, and -4.50 on Llama). Within PassageCount the card is the smallest term: removing aggregation costs 5.76, against 16.29 for the safeguards and 20.62 for source order (Table 3), so the card adds to whole-sentence retention rather than replacing it.

Q3. Classifier. We have not measured an error rate. A structural audit of the 1,451 queries assigned to count, existence, extreme or summary across LongBench and LongBench v2 found 196 of 417 count assignments (47%) coming from country, county or account substrings and 133 (32%) from code prefixes, while 7 of 601 existence assignments (1.2%) start with a yes/no question. Disabling the card bounds their effect: HotpotQA changes by -0.60 and QASPER by +0.80.

Q4. ZeroSCROLLS at 3K. The sentence you cite is gone. The column now reads 36.71 at 2K, behind RECOMP at 37.03, and 38.65 at 3K, the best value in that column; on Llama it is 26.75 and 28.99, third. Against EHPC the margin is 12.94 points on Qwen and 11.04 on Llama at 2K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.