- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is a characterization study of static, pre-tokenization input pruning for LLMs. It applies 15 fixed interventions to raw text: 11 linguistic rules (punctuation, subword, stopword/function-word, frequency, dependency-based, and morphological) and 4 random token-deletion controls. These are evaluated unchanged across 9 models (3B–14B) and 6 QA benchmarks, plus XQuAD in English, German and Turkish.
Critical strengths
A broad, carefully controlled evaluation. The same fixed rules are applied to 9 models from four families, 6 benchmarks and 3 languages. Every rule is compared against random deletion at a similar rate, and the paper uses paired tests with BH correction, bootstrap intervals and a reproducibility archive. This makes the descriptive findings, such as “P1 is free”, “D2 is the most efficient aggressive reduction” and “frequency filtering is worse than the trend”, reasonably trustworthy and useful as reference points.
Critical weaknesses
1 The central “coverage paradox” is mostly a consequence of the truncation protocol, and its causal support is weak. That pruning can help when the baseline must truncate is already known from LongLLMLingua. 2 The practical significance is limited. Retaining 63% of quality at 68% reduction is not usable in practice.
Questions for authors
Can you run a within-model context-length sweep on LongBench-v2 (the same model at several budgets) to test the coverage account directly?
3 comments
Response to Reviewer FQak (2/2)
W2: "Retaining 63% of quality at 68% reduction is not usable in practice." We agree that D2 is not meant as a deployment setting. It is a diagnostic stress condition that shows which structure carries the signal (Sec. 4.3). The practical results are different:
- Conservative rules are free. P1 removes 12.4% of the tokens. Across all 54 model-benchmark combinations, its quality changes by +0.14 points (95% CI [-0.59, 0.87]), so P1 is non-inferior for any margin of 0.5 points or more (App. J.4). It also keeps almost all of the benefit the document adds over the questions alone (+7.16 of +7.75 points; App. J.7) and keeps summarization quality (-0.12 [-0.45, 0.22] ROUGE-L; App. J.5).
- Static rules match a learned compressor at half the cost. With the same scope on LongBench-v2 (32k budget), static D2 scores 32.60% and LLMLingua-2 at 80% removal 31.81%, which is 0.80 points below D2 (difference [-4.72, 3.22], not significant), even though D2 removes more text (88.2% vs. 78.4%). The learned pipeline takes 12.3-13.4 s per item, D2 6.4 s, and D2 needs no compressor model (App. J.8).
- End-to-end efficiency. Averaged over six benchmarks, D2 gives a end-to-end speed-up and 40.9% lower peak memory; on QASPER and 89.7% (Sec. 4.3), pruning overhead included.
- The task decides. On long documents, D2 keeps 79.8% of the GovReport ROUGE-L at 86.3% removal, compared with 63% averaged over all benchmarks, which include short multiple-choice tasks where aggressive reduction is costly by design.
Our goal is not a new SOTA compressor but a map of which linguistic structures matter. In practice, conservative rules can be applied for free before any model computation, and aggressive reduction pays off where truncation would otherwise throw evidence away.
Summary of revisions. The within-model sweep is now the main controlled evidence (Sec. 4.4, Table 2); the positioning relative to LongLLMLingua is clarified (Sec. 2); CIs replace "no loss"; and the new App. J adds the Llama sweep, summarization, multiple masks, no-context, corrected-scoring and learned-compressor results.
We hope that the sweep you suggested and the revised text address your concerns. We would appreciate it if you would consider updating your score, and we are happy to discuss any further questions.
Dear Reviewer FQak,
Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.
Response to Reviewer FQak (1/2)
Thank you for the critical review. We ran the experiment you asked for (Sec. 4.4, Table 2, App. J.1).
Q: The same model at several LongBench-v2 context lengths. One model (Qwen3.5-9B), the same 503 items, four budgets, only the context pruned, questions and options unchanged (14,084 generations):
"Inputs truncated" is the share of items whose full input is longer than the budget and gets cut from the middle; LongBench-v2 documents are very long, so at 8k every item is cut. D2 accuracy hardly moves (31.0-32.6%), while the baseline loses 4.8 points as truncation grows. The D2 penalty therefore shrinks from 7.4 points at 64k (significant) to 2.8-3.8 points at 16k/8k (not significant), by 1.2 points per 10 points of extra truncation (95% CI [0.05, 2.33]). On the 108 long documents at 64k, it is no longer significant (-1.85 [-11.34, 6.78]).
W1: "The coverage effect is a consequence of the truncation protocol, and its causal support is weak." We agree that truncation drives the effect; that is the paper's point. The new sweep makes the causal case stronger: we now change the budget ourselves, on a fixed model and fixed items, so the evidence is interventional rather than correlational and no longer confounded by architecture. Random deletion at D2's reduction shows the same shrinking penalty, so the effect follows recovered coverage, not something specific to D2. (Llama-3.2-3B answers this benchmark close to chance at every budget; neither P1 nor D2 significantly lowers its accuracy up to 128k tokens; App. J.9.)
"Already known from LongLLMLingua." We credit LongLLMLingua and do not claim the coverage mechanism as new (Sec. 2, revised). What we add is different: