- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper examines whether a language model loss on the held out set provides a reliable basis to select context retention rules for the downstream tasks. The authors construct 21 rules and evaluate them on several decoder model using three loss corpora and four task settings. The main contribution is an empirical study which shows that a stable proxy ranking can lead to inferior task level choices.
Critical strengths
The two main strength are:
- their approach (selection-regret analysis) is more consequential than reporting low correlations alone
- the paper presents robustness checks supporting the disagreement (namely, ranking stability, paired uncertainty, stricter confidence thresholds is studied) which stregthen the paper results
Critical weaknesses
The main concern is that experiments select only among 128 context tokens within a 256-token window which significantly limits the paper applicability to real practical context compression methods usage.
There is also the concern about the evaluation methodology. When one compares loss across different datasets for the three lexical rules, the frequency table is also changed. As a consequence both the evaluation data and the actual behavior of the rules change simultaneously.
Questions for authors
Did you evaluated the lexical rules across datasets with the fixed frequency tables? (question about weakness 2)
3 comments
Thank you for the clarification.
Thank you for reading our response and for your points on context length and the frequency tables.
Thank you for the review. Both concerns are fair, and we ran an experiment for each.
Fixed frequency table (W2 and your question). We had not done this. Under PG-19 (books) and FineWeb (web pages) the three lexical rules that use a frequency table read one counted on that text, while the benchmarks read the table of WikiText-103 (Wikipedia articles). The text and the behavior of the rules changed together. We have now rescored PG-19 and FineWeb with WikiText-103's table on all six models, so that every loss and every benchmark reads the same table (Section 4, Appendix A.2).
The table does change where these rules rank. Even with the common table, though, the rule that keeps rare tokens ranks 7th or 8th of 21 under PG-19 and FineWeb loss on all six models, against 19th to 21st under WikiText-103 loss, so the difference between texts does not come from the table. Mean agreement with the four benchmarks is +0.59 for PG-19 (+0.50 with its own table) and +0.57 for FineWeb (+0.56), against +0.26 for WikiText-103. With one table everywhere, the losses on two different texts agree at +0.80, two benchmarks agree at +0.64, and the loss on ordinary text agrees with a benchmark at +0.47. Figure 1, which keeps each text's own table, gives +0.83, +0.64 and +0.44.
Context length (W1). You are right that 128 tokens is far from practice. We ran the 21 rules at 3840 tokens on two question-answering tasks that we built in the style of RULER (Appendix C.5). Each has 400 questions, and each question comes with a context of exactly 3840 tokens: the paragraphs that answer it, set among distractor paragraphs that do not. In the first task this is one SQuAD paragraph among other SQuAD paragraphs. In the second it is the two paragraphs of a HotpotQA question, which is written to need both, among more distractor paragraphs than HotpotQA itself provides (yes/no questions removed). As on our SQuAD ruler, a question counts as answered when every token of the answer is the model's top prediction. The loss is read on PG-19 and FineWeb, 512 windows each. WikiText-103 is not used there, because its test split is too short for 512 windows of 3840 + 256 tokens, but every ruler there reads WikiText-103's table.
The three small models are Qwen3-0.6B, Llama-3.2-1B and OLMo-2-1B. A task is used on a model only if, with the full context, the model answers at least 25% of its questions and at least 10 points more than the weakest of three simple reference rules, so that the rules have room to differ. Llama-3.2-1B answers 24% of the SQuAD-paragraph task, so the value between the two tasks is the mean over Qwen3-0.6B and OLMo-2-1B.
With half or one eighth kept on Qwen3-8B, and with half kept on the three small models, the loss agrees with the two tasks less than the tasks agree with each other. It agrees as well only with one eighth kept on the small models, where few questions are answered under any rule: even the best of the 21 rules answers only 7.5 to 13.5% of the questions there, against 25.2 to 41.5% with the whole context. This setting, the one nearest the long contexts and small budgets of published methods, goes against our finding at 128 tokens, and the paper now says so (Section 4). There the tasks reproduce their own ordering at only 0.63 to 0.92 across halves of their questions, against 0.99 or more for the loss, and part of the match may come from that, since low reliability lowers the agreement between two tasks more than that between loss and task. The low hit rate may also contribute; we have tested neither. On a needle-retrieval task in RULER's style, which we added, the loss also misranks the rules with half kept (+0.35 over all six models, with 34% of pairs reversed), but this task agrees just as little with the two question-answering tasks (+0.34).
At 128 tokens, keeping less does not remove the disagreement: with one quarter or one eighth kept, the loss agrees with a benchmark at +0.46 and +0.36 on average, against +0.58 and +0.48 between two benchmarks.
The paper now states its finding for 128 tokens, from one half down to one eighth kept. At 3840 tokens the same disagreement appears in three of the four settings, but not in all, so the paper makes no claim for long contexts in general (abstract, Section 8, Appendix C.5). A test that holds the tasks and the texts fixed across lengths is left for future work. Our rules are still applied as token dropping, and no released compression method was run.
Our comment to all reviewers summarizes what the paper claims and what changed in the revision.