acceptodds
Under review as a conference paper at ICLR 2027

Less Text, More Evidence: Static Pruning and the Coverage Paradox in Language Models

Abstract

Human skim reading suggests that linguistic information is distributed non-uniformly. We ask whether Large Language Models (LLMs) exhibit an analogous, statically identifiable selectivity: which linguistic structures in the input are load-bearing, and which can be removed before tokenization? We apply 15 fixed interventions-11 linguistically motivated pruning rules and four random token-deletion controls-to raw text, without LLM-based scoring or task-specific training, and evaluate them unchanged across nine LLMs and six benchmarks. Retaining only the subject-verb-object core of each sentence discards 68% of input tokens on average while retaining 63% of baseline quality; after adjusting for reduction rate, this rule retains the most quality among the linguistic rules on every evaluated model, whereas frequency-based filtering falls below the trend. The effect depends strongly on the task and the context regime. When inputs exceed the context window, pruning also changes which evidence reaches the model: on the longest LongBench-v2 documents, the pruning penalty observed on short documents disappears, and on identical items, the size of this reversal tracks how often each model must truncate the unpruned input (Spearman across models). We characterize this reversal as the coverage paradox and show that it depends jointly on recovered coverage and retained linguistic content. Our results provide a controlled map of which structures are removable and how these sensitivities transfer across models, tokenizers, and languages, and they imply that input reduction must be evaluated jointly with the context budget and truncation protocol, not by token count alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Pruning, Long-Context Efficiency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_FQakOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is a characterization study of static, pre-tokenization input pruning for LLMs. It applies 15 fixed interventions to raw text: 11 linguistic rules (punctuation, subword, stopword/function-word, frequency, dependency-based, and morphological) and 4 random token-deletion controls. These are evaluated unchanged across 9 models (3B–14B) and 6 QA benchmarks, plus XQuAD in English, German and Turkish.

Critical strengths

A broad, carefully controlled evaluation. The same fixed rules are applied to 9 models from four families, 6 benchmarks and 3 languages. Every rule is compared against random deletion at a similar rate, and the paper uses paired tests with BH correction, bootstrap intervals and a reproducibility archive. This makes the descriptive findings, such as “P1 is free”, “D2 is the most efficient aggressive reduction” and “frequency filtering is worse than the trend”, reasonably trustworthy and useful as reference points.

Critical weaknesses

1 The central “coverage paradox” is mostly a consequence of the truncation protocol, and its causal support is weak. That pruning can help when the baseline must truncate is already known from LongLLMLingua. 2 The practical significance is limited. Retaining 63% of quality at 68% reduction is not usable in practice.

Questions for authors

Can you run a within-model context-length sweep on LongBench-v2 (the same model at several budgets) to test the coverage account directly?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FQak (1/2)

Thank you for the critical review. We ran the experiment you asked for (Sec. 4.4, Table 2, App. J.1).

Q: The same model at several LongBench-v2 context lengths. One model (Qwen3.5-9B), the same 503 items, four budgets, only the context pruned, questions and options unchanged (14,084 generations):

BudgetInputs truncatedBaseline acc.D2 acc.D2 - baseline [95% CI]
8k100.0%34.7931.01-3.78 [-7.78, +0.20]
16k95.6%34.9932.21-2.78 [-7.09, +1.39]
32k77.9%38.7732.60-6.16 [-10.57, -1.79]
64k63.8%39.5632.21-7.36 [-12.04, -2.82]

"Inputs truncated" is the share of items whose full input is longer than the budget and gets cut from the middle; LongBench-v2 documents are very long, so at 8k every item is cut. D2 accuracy hardly moves (31.0-32.6%), while the baseline loses 4.8 points as truncation grows. The D2 penalty therefore shrinks from 7.4 points at 64k (significant) to 2.8-3.8 points at 16k/8k (not significant), by 1.2 points per 10 points of extra truncation (95% CI [0.05, 2.33]). On the 108 long documents at 64k, it is no longer significant (-1.85 [-11.34, 6.78]).

W1: "The coverage effect is a consequence of the truncation protocol, and its causal support is weak." We agree that truncation drives the effect; that is the paper's point. The new sweep makes the causal case stronger: we now change the budget ourselves, on a fixed model and fixed items, so the evidence is interventional rather than correlational and no longer confounded by architecture. Random deletion at D2's reduction shows the same shrinking penalty, so the effect follows recovered coverage, not something specific to D2. (Llama-3.2-3B answers this benchmark close to chance at every budget; neither P1 nor D2 significantly lowers its accuracy up to 128k tokens; App. J.9.)

"Already known from LongLLMLingua." We credit LongLLMLingua and do not claim the coverage mechanism as new (Sec. 2, revised). What we add is different:

  • Static, model-free rules. LongLLMLingua scores tokens with a model. We show that the effect appears even for fixed rules applied before tokenization, and we characterize it with random controls, nine models and a within-model sweep.
  • A concrete lesson for evaluation. The same rule on the same model and items costs 7.4 points at 64k but only 2.8 points at 16k. Compression results reported without context budget and truncation protocol are therefore hard to interpret.
  • Where structure matters beyond coverage. D2 beats random deletion on LooGLE (+4.17 [2.13, 6.43] ROUGE-L) and on GovReport, where we matched the removed share exactly (+2.83 [2.37, 3.29]; App. J.5, J.10).
AuthorsOpenReview ↗

Response to Reviewer FQak (2/2)

W2: "Retaining 63% of quality at 68% reduction is not usable in practice." We agree that D2 is not meant as a deployment setting. It is a diagnostic stress condition that shows which structure carries the signal (Sec. 4.3). The practical results are different:

  • Conservative rules are free. P1 removes 12.4% of the tokens. Across all 54 model-benchmark combinations, its quality changes by +0.14 points (95% CI [-0.59, 0.87]), so P1 is non-inferior for any margin of 0.5 points or more (App. J.4). It also keeps almost all of the benefit the document adds over the questions alone (+7.16 of +7.75 points; App. J.7) and keeps summarization quality (-0.12 [-0.45, 0.22] ROUGE-L; App. J.5).
  • Static rules match a learned compressor at half the cost. With the same scope on LongBench-v2 (32k budget), static D2 scores 32.60% and LLMLingua-2 at 80% removal 31.81%, which is 0.80 points below D2 (difference [-4.72, 3.22], not significant), even though D2 removes more text (88.2% vs. 78.4%). The learned pipeline takes 12.3-13.4 s per item, D2 6.4 s, and D2 needs no compressor model (App. J.8).
  • End-to-end efficiency. Averaged over six benchmarks, D2 gives a 2.28×2.28\times end-to-end speed-up and 40.9% lower peak memory; on QASPER 5.44×5.44\times and 89.7% (Sec. 4.3), pruning overhead included.
  • The task decides. On long documents, D2 keeps 79.8% of the GovReport ROUGE-L at 86.3% removal, compared with 63% averaged over all benchmarks, which include short multiple-choice tasks where aggressive reduction is costly by design.

Our goal is not a new SOTA compressor but a map of which linguistic structures matter. In practice, conservative rules can be applied for free before any model computation, and aggressive reduction pays off where truncation would otherwise throw evidence away.

Summary of revisions. The within-model sweep is now the main controlled evidence (Sec. 4.4, Table 2); the positioning relative to LongLLMLingua is clarified (Sec. 2); CIs replace "no loss"; and the new App. J adds the Llama sweep, summarization, multiple masks, no-context, corrected-scoring and learned-compressor results.

We hope that the sweep you suggested and the revised text address your concerns. We would appreciate it if you would consider updating your score, and we are happy to discuss any further questions.

Dear Reviewer FQak,

Thank you again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are happy to answer any remaining questions or clarify further points.

Reviewer_CWC1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether LLM input contains statically identifiable linguistic redundancy that can be removed before tokenization. It evaluates 15 fixed interventions across nine LLMs and six benchmarks, without using the target LLM to determine token importance.
The central result is the coverage paradox, where pruning can remove local information while simultaneously allowing previously truncated evidence to enter the context. On LongBench-v2, D2's relative performance improves substantially from short to long documents, and the size of this reversal correlates strongly with models' baseline truncation incidence. The structural ranking transfers strongly across models, although context-sensitive long-document results are less stable. The paper argues that input compression should be evaluated jointly with task requirements, context-window size, and truncation policy, rather than judged solely by percentage of tokens removed.

Critical strengths

The coverage is very broad with 9 models, 6 benchmarks, and 15 interventions, along with multiple tokenizers and controls. The coverage paradox is a valuable methodological observation, as it explains why pruning can improve accuracy, even though information is being deleted. the relevant counterfactual may be a baseline that has already lost large portions of the document through truncation. Random deletion controls further show that recovered coverage itself explains a large portion of the effect. The paper also reports inference-time and memory effects, includes pruning overhead. D2 yields meaningful average speed and memory improvements.

Critical weaknesses

All evaluated tasks are variants of question answering. It is therefore unclear whether the same structural hierarchy holds for other tasks like summarization, generation, translation, etc. Random-deletion controls use only one mask per item, leaving mask-level variability unmeasured. Multiple random masks would make the comparison substantially stronger.

Questions for authors

What would happen if you expand beyond QA task? And what would happen if you use multiple random masks per reduction level?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CWC1

Thank you for the positive assessment and for the two concrete suggestions. We followed both (App. J.2, J.5, J.10).

W1/Q: Beyond QA. We added GovReport summarization: 200 documents, Qwen3.5-9B, a 16k-token budget, summaries of up to 512 tokens, ROUGE-L, and three random masks per document.

ConditionRemovalROUGE-LDifference to baseline [95% CI]
Baseline0%20.59–
P19.6%20.47-0.12 [-0.45, +0.22]
S222.3%20.18-0.41 [-0.81, -0.01]
Random 20%20.2%20.23-0.36 [-0.66, -0.06]
D286.3%16.43-4.16 [-4.75, -3.60]
Random, same removal as D286.3%13.59-7.00 [-7.68, -6.32]
LLMLingua-2 (80%)77.7%18.02-2.57 [-2.98, -2.16]
  • The structural hierarchy carries over to summarization. As in QA, P1 is nearly free (the CI rules out losses above half a point), and S2 and random 20% deletion lose less than half a point.
  • Structure matters beyond length. For the cleanest test, we matched random deletion to D2 document by document, so both remove exactly 86.3% of the text. D2 is still clearly better: +2.83 [2.37, 3.29] ROUGE-L (p<0.001). It keeps 79.8% of the baseline ROUGE-L, random deletion only 66.0%.
  • Learned reference. LLMLingua-2 keeps more ROUGE-L at a lower removal rate but needs a compressor model; the static rules need none.

ROUGE-L measures overlap rather than factual consistency; translation and open-ended generation remain future work (Sec. 5).

W2/Q: Multiple random masks. We now use three masks per item (seeds 42-44) in three studies: the LongBench-v2 budget sweep (App. J.2), the matched GovReport run above, and a new study at 20/40/60/80% deletion on SQuAD v2, LongBench-v2, LooGLE and QASPER (104,524 generations; App. J.10). Single items do vary between masks, but benchmark scores are stable:

BenchmarkStd. dev. across masksD2 - random 80% [95% CI]
SQuAD v2 (F1)≤0.38\le 0.38-0.06 [-0.77, +0.67]
LongBench-v2 (acc.)≤1.83\le 1.83-2.78 [-8.37, +2.59]
LooGLE (ROUGE-L)≤0.82\le 0.82+4.17 [+2.13, +6.43]
QASPER (F1)≤1.06\le 1.06-0.90 [-2.68, +0.84]

So single-mask results are stable at the resolution of our comparisons. D2 is never significantly worse than random deletion, even though it removes 4.4–7.9 points more of the text on SQuAD v2, LongBench-v2 and QASPER. Averaged over the four benchmarks, D2 keeps 65.2% of baseline quality at 84.8% removal, versus 64.4% at 80.3% for random deletion: the same quality at a higher reduction. Where answers depend on local detail, D2 is clearly better (LooGLE +4.17; GovReport at matched removal +2.83).

We hope that the summarization experiment and the multi-mask results address both of your suggestions. We would appreciate it if you would consider updating your score, and we are glad to clarify anything that remains open.

Dear authors,

Thanks for providing clarifications. I will stick with my current rating of weak acceptance.

Thank you for your reply and for your positive assessment. Should any question remain, we are happy to clarify it during the discussion period.

Reviewer_gW6BOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

To study the statically identifiable selective of linguistic structures, the paper evaluates eleven linguistic pruning rules and four random controls across multiple models and benchmarks. It argues that pruning can offset information loss by exposing otherwise truncated text. The descriptive evidence is broad, but the proposed mechanisms require better controls and corrected evaluation.

Critical strengths

The paper has broad empirical coverage: four model families, several QA tasks, and multilingual tests. Statistical results with classical statistics-style tests (paired tests, multiple-testing correction, bootstrap uncertainty, and absolute counts) are reported. The work also has useful practical framing: it distinguishes characterization from state-of-the-art compression, includes LLMLingua-2 references, and measures pruning overhead and total latency.

Critical weaknesses
  1. The coverage mechanism remains confounded. The correlation of 0.93 spans eight models with only five distinct truncation rates (Appendix D). Architecture varies alongside context budget.

  2. It's possible that pruning also changes the task. LongBench-v2 pruning affects context, questions, and options (Table 7), potentially removing task-defining negation or distinctions. This complicates attribution to evidence coverage and comparison with the context-only LLMLingua-2 reference.

  3. Quality preservation is not established. Nonsignificant degradation does not establish equivalence, and most individual gains do not survive correction.

  4. While I like the motivations and results of the paper, the problems and methods seem more like traditional linguistics and statistics. A different conference/journal on linguistics might be a better venue for this paper.

Questions for authors
  1. Can you report a within-model budget sweep on identical items? Does the coverage reversal persist when only context is pruned, questions/options stay intact, and context budget varies within one model?

  2. After correcting output normalization and dependency labels, which conclusions remain? Please include absolute paired differences, no-context controls, and non-inferiority intervals for preservation claims.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gW6B (1/2)

Thank you for the careful reading and for the two concrete experimental questions. We ran both.

W1/Q1: Architecture changes together with the budget (Sec. 4.4, Table 2, App. J.1). You are right that the cross-model correlation (ρs=0.93\rho_s=0.93) cannot separate the two. So we ran the design you suggested: one model (Qwen3.5-9B), the same 503 LongBench-v2 items, four context budgets, and only the context pruned (instructions, questions and options untouched).

BudgetInputs truncatedBaseline acc.D2 acc.D2 - baseline [95% CI]
8k100.0%34.7931.01-3.78 [-7.78, +0.20]
16k95.6%34.9932.21-2.78 [-7.09, +1.39]
32k77.9%38.7732.60-6.16 [-10.57, -1.79]
64k63.8%39.5632.21-7.36 [-12.04, -2.82]

"Inputs truncated" is the share of items whose full input is longer than the budget and gets cut from the middle. LongBench-v2 documents are very long, so at 8k every item is cut.

  • The coverage pattern holds with the architecture fixed. D2 accuracy barely changes, while the baseline loses 4.8 points as truncation grows. The D2 penalty is significant at 64k but not at 16k or 8k.
  • The trend is significant: the penalty shrinks by 1.2 points per 10 points of extra truncation (95% CI [0.05, 2.33]). The direct 8k-vs-64k comparison points the same way (+3.58 [-0.81, 8.05]).
  • Does the reversal persist? The paper's robust finding on long documents is that the penalty disappears (Sec. 4.4), and that holds here too: it is no longer significant at 8k/16k, and on the 108 long documents at 64k it is -1.85 [-11.34, 6.78]. Over all items D2 stays below the baseline, so we call this a shrinking penalty rather than a gain.
  • Coverage drives it. Random deletion at D2's reduction shows the same shrinking penalty (within 1.0-1.7 points of D2 at every budget), and P1 stays within 1.6 points of the baseline.

Results are identical under four answer-extraction rules. We also ran Llama-3.2-3B up to 128k tokens (App. J.9). It answers this benchmark close to chance at every budget, which leaves little room for any effect; neither P1 nor D2 significantly lowers its accuracy. The revision now presents the within-model sweep as the main controlled evidence and the cross-model correlation as supporting description.

W2: Pruning on LongBench-v2 also changed the questions and options. Good catch, thank you. Applied to all 503 items, D2 removes a negation or contrast cue from 127 questions (25.2%), merges two options on 73 items (14.5%) and empties an option on 14 (2.8%) (App. J.3). All new LongBench-v2 experiments therefore prune only the context, and they show the same picture as the paper: P1 is nearly free, and the cost of D2 depends on the budget. This also makes the LLMLingua-2 comparison fair: with the same scope, LLMLingua-2 at 80% removal scores 31.81%, which is 0.80 points below static D2 (32.60%; difference [-4.72, 3.22], not significant; App. J.8). MMLU-Pro and ARC-Challenge always kept their options intact.

AuthorsOpenReview ↗

Response to Reviewer gW6B (2/2)

W3: Non-significance is not equivalence. Agreed. We now report paired differences with 95% CIs throughout, so the size of any possible loss is visible:

ContrastDifference [95% CI]
P1, all 54 model–benchmark combinations+0.14 [-0.59, +0.87]
P1, within-model sweep, 8k-64k-1.59 to +0.40, all CIs include 0
P1, GovReport summarization (ROUGE-L)-0.12 [-0.45, +0.22]
D2, sweep at 8k / 64k-3.78 [-7.78, 0.20] / -7.36 [-12.04, -2.82]

For P1, the one-sided 95% lower bound over all 54 combinations is -0.48, so P1 is non-inferior for any margin of 0.5 points or more. On summarization, the CI rules out losses larger than 0.45 ROUGE-L points. We no longer claim individual long-document gains; in line with the corrected tests, the paper reports that the penalty disappears (Sec. 4.4).

W4: Venue fit. The rules are deliberately simple, model-free probes of how LLMs use their input. The main lesson is about ML evaluation: on the same model and items, the same rule costs 7.4 points at 64k but only 2.8 points at 16k. Prompt- and KV-cache-compression results are therefore hard to interpret without the context budget and the truncation protocol. The introduction now says this explicitly.

Q2: What remains after the corrections? All main conclusions.

  • Output normalization (App. J.4). After re-scoring all 3,788,830 archived predictions without end-of-turn tokens, only 1 of 108 P1/D2 contrasts changes sign (Phi-4, SQuAD v2: −0.18→+0.20-0.18 \to +0.20); multiple-choice accuracy is unchanged. P1 moves from +0.04 to +0.14 [-0.59, 0.87], D2 from -16.40 to -17.96.
  • XQuAD (App. G). Re-scoring raises absolute scores (Qwen3.5-9B English baseline: exact match 0→70.40 \to 70.4, F1 66.1→82.766.1 \to 82.7). Because the tokens affected baseline and interventions alike, 131 of 153 rank positions stay the same. All XQuAD tables and figures now use the re-scored values.
  • Dependency labels (App. J.6). With the German label scheme, D1/D2 keep 13.5%/17.1% of the German context (English: 23.9%/13.0%; Turkish: 14.3%/19.7%). This points to a failure of rule transfer, not a property of German.
  • No-context control (App. J.7). The document adds +7.75 [2.43, 12.99] points over questions and options alone (p = 0.005), so LongBench-v2 does measure context use, and P1 keeps almost all of it (+7.16 [2.18, 12.16]). At 80-90% removal, neither D2 (+1.59 [-2.87, 5.99]) nor LLMLingua-2 (+0.80 [-3.76, 5.38]) keeps a significant benefit over this control. This is why the paper treats D2 as a diagnostic stress condition and not as an operating point.

In short: conservative rules such as P1 are nearly free, with tight CIs; D2 is the most informative aggressive rule; and the cost of pruning depends on the context budget, now shown within one model on identical items.

We hope that the within-model sweep and the corrected analyses address both of your questions. If so, we would appreciate it if you would consider updating your score, and we are happy to answer any further questions.

Thank you for your rebuttal, which basically address my questions. I will raise my rating to 4.

Thank you for raising your rating! Your suggested within-model budget sweep is now the main controlled evidence of the paper (Sec. 4.4), and we appreciate your constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new experiments and revised manuscript

We thank all reviewers for their thoughtful feedback. We were glad to see the breadth of the study (nine LLMs, six benchmarks, 15 fixed interventions), the paired statistics and the random-deletion controls appreciated. We ran every experiment that was asked for. All changes in the revised PDF are marked in blue; the new results are in Sec. 4.4 (Table 2) and in a new App. J. A short overview:

  • Budget sweep within one model (gW6B, FQak; Sec. 4.4, App. J.1). We fixed the model (Qwen3.5-9B) and the items (all 503 LongBench-v2 items), pruned only the context, and varied the budget from 8k to 64k tokens. D2 accuracy hardly moves (31.0-32.6%), while the dense baseline loses 4.8 points as more of its input gets truncated. The D2 penalty therefore shrinks from 7.4 points at 64k to 2.8-3.8 points at 8k/16k, by 1.2 points for every 10 points of extra truncation (95% CI [0.05, 2.33]). Since model, tokenizer and items are fixed, this cannot come from differences between models.
  • Second model (App. J.9). Llama-3.2-3B, run up to 128k tokens, answers LongBench-v2 close to chance at every budget. Within that range, neither P1 nor D2 significantly lowers its accuracy, even though D2 removes 88% of the text.
  • Summarization (CWC1; App. J.5). On GovReport, P1 changes ROUGE-L by only -0.12 [-0.45, 0.22]. Against random deletion that removes exactly as much text as D2 (86.3%), D2 is clearly better: +2.83 [2.37, 3.29] ROUGE-L.
  • Multiple random masks (CWC1; App. J.2, J.10). Three masks per item on four benchmarks (104,524 generations). Benchmark scores change by a standard deviation of at most 1.8 points across masks, so single-mask results are stable. D2 is never significantly worse than random deletion and is clearly better on LooGLE (+4.17 [2.13, 6.43] ROUGE-L).
  • Corrected scoring (gW6B; App. J.4). After removing end-of-turn tokens and re-scoring all 3,788,830 predictions, only 1 of 108 P1/D2 contrasts changes sign. Across all 54 model-benchmark combinations, P1 changes quality by +0.14 [-0.59, 0.87] points.
  • No-context control (gW6B; App. J.7). The document adds +7.75 [2.43, 12.99] points over the questions and options alone, and P1 keeps almost all of that (+7.16 [2.18, 12.16]).
  • Learned compressor with the same scope (App. J.8). When only the context is compressed, LLMLingua-2 at 80% removal scores 31.81%, slightly below static D2 (32.60%), and takes about twice as long per item.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.