acceptodds
Under review as a conference paper at ICLR 2027

The Ruler Agrees With Itself, Not With the Task: Auditing Held-Out Loss as an Instrument for Context Selection

Abstract

When a language model's context is too long to keep whole, a selection rule decides which tokens stay. Such rules are often compared by held-out loss, which is cheap, label-free and precise. We ask whether the ordering it produces is the ordering a task produces. It is not, and where the two differ the loss is confident. We score twenty-one selection rules we constructed, each keeping half of a context's tokens, applied as token dropping on six frozen decoders and read on three loss corpora and four task benchmarks. Every ruler, a corpus or benchmark read with its own metric, reproduces its own ordering across halves of its data at a mean rank correlation of 0.956 to 0.996 corrected to full length; the loss corpora agree with one another at 0.83 on average, and a loss corpus with a benchmark at 0.44. Two instruments this reliable and this far apart measure different things. The disagreement sits at the top of the loss ranking: the loss prefers rules that keep recent or heavily attended tokens; the CBT benchmarks usually put a rule that keeps rare tokens first, LAMBADA usually second, and SQuAD's usual leader keeps the earliest tokens. It is not a matter of resolution: of the rule pairs that the WikiText-103 loss and a benchmark both order confidently, the benchmark reverses 30–44% (16–36% under the other corpora). Training a small selector on a frozen decoder against the loss does not remove the disagreement: on most task readings the selector stays in the rank band the loss gives it, but first place is decided by the task, and on CBT a rule the loss ranks in its bottom three usually beats the selector by point estimate. The obvious repair, a key-token gate that restricts the loss to tokens needing more than a short recent window, on average matches or modestly beats the plain loss only when that window sits well below the budget; tied to the budget, it measures the budget. We argue not for abandoning the loss but for auditing it before it chooses: the proxy's reliability, the target's, then their correlation read beside both.

Primary area:
datasets and benchmarks
Keywords:
evaluation methodology, measurement validity, perplexity, language model evaluation, context selection, context compression, token dropping, reliability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_L98iOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines whether a language model loss on the held out set provides a reliable basis to select context retention rules for the downstream tasks. The authors construct 21 rules and evaluate them on several decoder model using three loss corpora and four task settings. The main contribution is an empirical study which shows that a stable proxy ranking can lead to inferior task level choices.

Critical strengths

The two main strength are:

  1. their approach (selection-regret analysis) is more consequential than reporting low correlations alone
  2. the paper presents robustness checks supporting the disagreement (namely, ranking stability, paired uncertainty, stricter confidence thresholds is studied) which stregthen the paper results
Critical weaknesses

The main concern is that experiments select only among 128 context tokens within a 256-token window which significantly limits the paper applicability to real practical context compression methods usage.

There is also the concern about the evaluation methodology. When one compares loss across different datasets for the three lexical rules, the frequency table is also changed. As a consequence both the evaluation data and the actual behavior of the rules change simultaneously.

Questions for authors

Did you evaluated the lexical rules across datasets with the fixed frequency tables? (question about weakness 2)

3 comments

Thank you for the review. Both concerns are fair, and we ran an experiment for each.

Fixed frequency table (W2 and your question). We had not done this. Under PG-19 (books) and FineWeb (web pages) the three lexical rules that use a frequency table read one counted on that text, while the benchmarks read the table of WikiText-103 (Wikipedia articles). The text and the behavior of the rules changed together. We have now rescored PG-19 and FineWeb with WikiText-103's table on all six models, so that every loss and every benchmark reads the same table (Section 4, Appendix A.2).

The table does change where these rules rank. Even with the common table, though, the rule that keeps rare tokens ranks 7th or 8th of 21 under PG-19 and FineWeb loss on all six models, against 19th to 21st under WikiText-103 loss, so the difference between texts does not come from the table. Mean agreement with the four benchmarks is +0.59 for PG-19 (+0.50 with its own table) and +0.57 for FineWeb (+0.56), against +0.26 for WikiText-103. With one table everywhere, the losses on two different texts agree at +0.80, two benchmarks agree at +0.64, and the loss on ordinary text agrees with a benchmark at +0.47. Figure 1, which keeps each text's own table, gives +0.83, +0.64 and +0.44.

Context length (W1). You are right that 128 tokens is far from practice. We ran the 21 rules at 3840 tokens on two question-answering tasks that we built in the style of RULER (Appendix C.5). Each has 400 questions, and each question comes with a context of exactly 3840 tokens: the paragraphs that answer it, set among distractor paragraphs that do not. In the first task this is one SQuAD paragraph among other SQuAD paragraphs. In the second it is the two paragraphs of a HotpotQA question, which is written to need both, among more distractor paragraphs than HotpotQA itself provides (yes/no questions removed). As on our SQuAD ruler, a question counts as answered when every token of the answer is the model's top prediction. The loss is read on PG-19 and FineWeb, 512 windows each. WikiText-103 is not used there, because its test split is too short for 512 windows of 3840 + 256 tokens, but every ruler there reads WikiText-103's table.

3840 tokensLoss and taskThe two tasksReversed, of the pairs both order with confidence (SQuAD-paragraph task, HotpotQA task)
Three small models, half kept+0.67+0.8511%, 13%
Three small models, one eighth kept+0.73+0.694%, 4%
Qwen3-8B, half kept+0.38+0.8323%, 28%
Qwen3-8B, one eighth kept+0.62+0.6918%, 16%

The three small models are Qwen3-0.6B, Llama-3.2-1B and OLMo-2-1B. A task is used on a model only if, with the full context, the model answers at least 25% of its questions and at least 10 points more than the weakest of three simple reference rules, so that the rules have room to differ. Llama-3.2-1B answers 24% of the SQuAD-paragraph task, so the value between the two tasks is the mean over Qwen3-0.6B and OLMo-2-1B.

With half or one eighth kept on Qwen3-8B, and with half kept on the three small models, the loss agrees with the two tasks less than the tasks agree with each other. It agrees as well only with one eighth kept on the small models, where few questions are answered under any rule: even the best of the 21 rules answers only 7.5 to 13.5% of the questions there, against 25.2 to 41.5% with the whole context. This setting, the one nearest the long contexts and small budgets of published methods, goes against our finding at 128 tokens, and the paper now says so (Section 4). There the tasks reproduce their own ordering at only 0.63 to 0.92 across halves of their questions, against 0.99 or more for the loss, and part of the match may come from that, since low reliability lowers the agreement between two tasks more than that between loss and task. The low hit rate may also contribute; we have tested neither. On a needle-retrieval task in RULER's style, which we added, the loss also misranks the rules with half kept (+0.35 over all six models, with 34% of pairs reversed), but this task agrees just as little with the two question-answering tasks (+0.34).

At 128 tokens, keeping less does not remove the disagreement: with one quarter or one eighth kept, the loss agrees with a benchmark at +0.46 and +0.36 on average, against +0.58 and +0.48 between two benchmarks.

The paper now states its finding for 128 tokens, from one half down to one eighth kept. At 3840 tokens the same disagreement appears in three of the four settings, but not in all, so the paper makes no claim for long contexts in general (abstract, Section 8, Appendix C.5). A test that holds the tasks and the texts fixed across lengths is left for future work. Our rules are still applied as token dropping, and no released compression method was run.

Our comment to all reviewers summarizes what the paper claims and what changed in the revision.

Thank you for the clarification.

Thank you for reading our response and for your points on context length and the frequency tables.

Reviewer_xhjDOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

In this paper, the authors evaluate 21 “rules” for long-context selection and evaluate held-out loss on them across 7 datasets, 3 on loss and 4 on task performance. Their results show that the order of these rules is different across the datasets.

Critical strengths

Auditing commonly used methods is important, and the scope of the experiments is ambitious.

Critical weaknesses
  1. Clarity: The paper is very hard to read, uses unnecessarily complex sentence structure, and has a strong AI cadence (although AI translation is acknowledged).
  2. (This weakness might be due to my misunderstanding and lack of expertise on this topic; see Questions.) Comparing tasks and loss separately: to see if the tokens that minimize the loss are beneficial for the task, shouldn’t the same dataset be used?
Questions for authors

Why do you assume that the best rules selected with loss on, say, FineWeb, should be the same as the rules selected by task performance on, say, LAMBADA? You explain this on line 098, but isn’t minimizing uncertainty for the prompt different from performing well on a task?

My expertise is not well suited for this paper; hence, if the pointed weaknesses are due to this, I’ll take that into account in the recommendation.

3 comments

Thank you for the review. Both of your points came from our writing, not from a lack of expertise.

The difference you point to is the paper's finding. We agree that predicting text well differs from doing well on a task, and the paper does not assume otherwise. The assumption belongs to the step we examine: published methods that compress a model's context have settled some design choices by held-out loss (Section 2), and using such a choice on a task assumes that loss and task order the options alike. We test this assumption and find that it does not hold for contexts of up to 128 tokens. Line 098 stated the logic of the test, but it read as our own assumption. The introduction now says: "We audit that use: a rule that does well on the loss is then used on a task. That choice is sound only if loss and task order the rules alike."

Same dataset. You are right that the cleaner comparison keeps the dataset fixed. The submission compared across datasets because the step we audit does: rules are ranked by loss on ordinary text (Wikipedia articles, books or web pages) and then used on a task. Two same-dataset comparisons follow.

  1. Same items, loss of the correct answer (in the submission). For each item we also record the loss of the correct answer: LAMBADA's final word, CBT's correct candidate, SQuAD's answer span. This loss and accuracy order the rules almost alike: +0.91 (LAMBADA), +0.97 (CBT-NE and CBT-CN) and +0.86 (SQuAD). But this loss needs the answers, and with the answers one can measure accuracy directly.

  2. Same text, no labels (new, Appendix C.4). We treat each benchmark's own text like any other text: LAMBADA's passages without the final word, the books the CBT items come from (two of the five books and most of a third), and SQuAD's paragraphs without the questions. We cut it into windows and measure the ordinary held-out loss: how well the model predicts the next 128 tokens after a rule has dropped half of the 128 tokens before them. No labels are used. Rank correlation with accuracy, averaged over the six decoders:

BenchmarkLoss on its own textLoss on WikiText-103 (Wikipedia articles)
LAMBADA+0.67+0.42
CBT-NE+0.65+0.25
CBT-CN+0.64+0.25
SQuAD+0.27+0.11

On all four benchmarks the own-text loss agrees better with accuracy than the WikiText-103 loss does, so part of the gap between loss and accuracy comes from the text. But a gap remains on each, and three observations narrow down why.

  • It is not noise. With the data split in half, the own-text loss reproduces its ordering at 0.996 or more, and the four benchmarks reproduce theirs at 0.956 to 0.993. If the two measured the same thing, they would agree almost that well. Instead they agree at +0.64 to +0.67 on LAMBADA and CBT, about as well as two different benchmarks agree (+0.64 on average), and at +0.27 on SQuAD. Of the rule pairs that both order with confidence, 15 to 37% are reversed.
  • Matching the text does not remove it. Read as running text, a benchmark's own text is sometimes not even the best match. LAMBADA's accuracy agrees better with the loss on the CBT books (+0.81) and on SQuAD's paragraphs (+0.70) than with the loss on its own passages (+0.67), and SQuAD's accuracy agrees with the loss on the CBT books (+0.31) about as well as with that on its own paragraphs (+0.27).
  • It fits the fact that the two score different things. The loss scores every token of running text, and the rules it ranks highest keep the most recent tokens or those that the most recent positions attend to. A benchmark scores only its answer, and each needs different context tokens for it: on CBT, whose answers are names or common nouns, the rule that keeps rare tokens usually ranks first, and on SQuAD the rule that keeps the earliest tokens. LAMBADA, whose answer is a passage's last word, is closest to what the loss scores, and its usual leader is one of the rules the loss ranks highest (Section 4). We have not tested this directly: the own-text windows also differ from the items in other ways, and the comparison cannot tell which difference matters.

Without labels, then, keeping the dataset fixed narrows the gap but does not close it. Our claim concerns the loss used to choose rules, which is measured on running text. A loss measured on a task's own inputs, in the task's format, is a different measurement, and we make no claim about it.

Readability. You are right, and we have revised the paper. The introduction now opens with two paragraphs of background and explains its main terms at first use, and long sentences are split. We also added explanations and polished the writing throughout.

Our comment to all reviewers summarizes what the paper claims and what changed in the revision. We would be grateful if you could tell us whether any part of our answer is still unclear.

Reviewer_xhjDOpenReview ↗

Response to Authors

Thank you for your response. I see you have added many explanations and clarifying sentences to the paper, making the clarity better. The AI cadence still makes the paper a bit difficult at times, so I will revise my clarity score by one point. I also thank you for running the experiment on the same text to address my concerns. As this paper does not fall under my expertise, I am not able to fully evaluate the new experiments (above or those added to the manuscript). Hence, I will also update my score to 2, but note my expertise as a limitation that can be taken into account in the final recommendation.

Thank you for reading the revision and for suggesting the experiment on the same text. We will keep simplifying the writing.

Reviewer_2HVwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the evaluation of context-selection rules, which decide which tokens to keep in case of too long LM's context. The authors analyze whether the ordering of those rules under held-out LM loss on a frozen model (by which such rules are often compared) agrees with the ordering under a downstream task. The experimental setup includes 21 selection rules (built by the authors), 6 frozen decoders, and 7 "rulers" (evaluation sets read with their own metric: 3 text corpora read by loss and 4 downstream tasks read by accuracy). The experiments demonstrate that, although every ruler is reliable, loss and task rankings only moderately correlate on average, and that the disagreement is not removed either by training a selector against the loss or by a LongPPL-style gate. As the main recommendation, the authors highlight the importance of auditing a proxy's and a target's reliability before trusting their correlation to choose among methods.

Critical strengths
  • The paper addresses an important practical question: whether held-out loss, which is often used to compare context-selection methods, ranks them the same way downstream tasks would do
  • The experimental setup includes a sufficient range of hand-crafted rules that read different signals, as well as models of different sizes and families. The design also includes controls: sign-reversed rules, a random rule, and a noise-gradient control for the trained selector.
Critical weaknesses
  • The paper is hard to follow, many sentences are convoluted and hard to parse. A lot of space is spent restating scores that are already shown in the figures (e.g., the values of Figure 1 in L247–259, Figure 4 in L.369–381, etc), and the headline numbers are repeated several times across the abstract, introduction and body. I'd recommend using this space instead to move some content from the appendix to the main text, since almost all main claims currently refer to the appendix for clarification.
  • My main concern is that the central claim of the paper, that held-out loss misranks selection rules relative to downstream tasks, is not supported by the evidence: the main finding concerns the evaluation text rather than the loss itself. When the same benchmark is scored by loss or by accuracy, the rankings closely agree on all benchmarks, and the authors themselves conclude that the difference is mostly the text scored, not the metric. The fact that loss on a generic dataset does not predict the ranking on a specific task is largely expected and doesn't require additional proving, and I don't see it as evidence that held-out loss is a poor instrument on its own for this use, or that the disagreement stems from the loss only (as the title and abstract suggest)
  • The main experiments use only a 128-token context, which is highly limited and not comparable to the practical context-selection setups that operate on thousands of tokens with much smaller budgets. In the only longer-context experiment (C = 512), the WikiText–CBT disagreement narrows sharply when 1/8 of the context is kept: agreement rises to +0.71/+0.67 (compared to +0.25 at C = 128). This is the setting closest to practice, but this finding is mentioned only briefly in the introduction; I believe it requires a more detailed discussion, as it questions how far the main conclusions generalise.
  • The task side is narrow. All four benchmarks require locating a single word or span, so the observed preference for keeping rare tokens may be specific to this choice of benchmarks. Also, the benchmarks do not agree with each other either (SQuAD at 0.40–0.60 with the others), so it is not clear which "task" the loss is expected to predict. I'd suggest evaluating on benchmarks commonly used for context selection and KV-cache compression, such as LongBench or the RULER benchmark.
  • The headline results rely on the corpus that transfers worst. Averaged over the four tasks, agreement with the task rankings is 0.26 for WikiText-103, but 0.50 for PG-19 and 0.56 for FineWeb (based on Figure 1). PG-19 and FineWeb agree with LAMBADA more than the tasks agree with each other on average (0.71 and 0.74 vs 0.64). Given that PG-19 and FineWeb transfer substantially better, I don't understand why the opening example, the main reversal counts, and the trained selector all use WikiText-103.
  • The importance of repair experiment is highly questionable. Simply switching the corpus from WikiText-103 to FineWeb improves agreement much more (0.26 -> 0.56, based on Figure 1) than the gate does (0.26 -> 0.34), but this option is not considered as an alternative repair.
Questions for authors

See Weaknesses above

5 comments

Thank you for the detailed review. We answer the weaknesses in order, in three comments.

W1. Presentation. All four points are right, and we have rewritten the main text.

  • Sentences. Long sentences are split throughout the main text, and most headings in the results sections now state their findings.
  • Scores already in figures. The text repeats fewer values of Figure 1 and gives the ranks of Figure 4 as ranges and counts rather than model by model. The three averages of Figure 1 stay in Section 4, because the argument rests on them.
  • Repeated headline numbers. The introduction now gives them once, in one paragraph, and states its contributions in words.
  • Content from the appendix. Three explanations moved into the main text: what the rules keep, what each benchmark asks, and how many reversals chance alone would give (Sections 3.1, 3.3 and 5). The account of how reliability is computed is longer. The main text now states each main claim, with its numbers at 128 tokens, and the appendix holds the details and the numbers for longer contexts.

W2. The central claim. You are right that the submission overstated its claim.

What we got wrong. Two things. First, the abstract presented the finding as a property of the loss itself, which our evidence does not show. Second, the sentence you cite, that what separates loss from task "is mostly the text scored, not the metric", said more than we had measured. The comparison behind it keeps the benchmark fixed and changes only the metric: read on a benchmark's own labeled answers, loss and accuracy order the rules almost alike (+0.86 to +0.97). That shows that the metric is not the issue, but not how much of the gap comes from the text. Both are corrected (abstract, Section 4). The claim now concerns the loss as published methods use it to choose among designs: read on ordinary running text, with the choice then used on a task. We kept the title because a ruler, as the paper defines it, is an evaluation set read with its own metric: the title's ruler is a corpus read by loss, not the loss alone.

The loss on the benchmark's own text. We also read the same loss on each benchmark's own text, as running text and without labels: LAMBADA's passages, the CBT books and SQuAD's paragraphs (new experiment, Section 4). Rank correlation with the benchmark's accuracy, averaged over the six models:

BenchmarkLoss on its own textLoss on WikiText-103 (Wikipedia articles)
LAMBADA+0.67+0.42
CBT-NE+0.65+0.25
CBT-CN+0.64+0.25
SQuAD+0.27+0.11

Compared with WikiText-103, the disagreement shrinks on every benchmark but does not go away. On LAMBADA and CBT the agreement rises to between +0.64 and +0.67, about as high as between two different benchmarks (+0.64 on average), and it stays below the +0.86 to +0.97 of the labeled loss. On SQuAD it is +0.27. Of the rule pairs that both sides order with confidence, 15 to 37% are still reversed. Our claim concerns the loss used to choose rules, which is read on running text. A loss read on a task's own inputs, in the task's format, is a different measurement, and we make no claim about it.

"Largely expected." We agree that a loss on generic text need not predict a specific task. The measurement is still worth having, for two reasons. First, published work does choose by loss on ordinary text: StreamingLLM's number of sink tokens and TOVA's choice between its per-head and per-layer variants were decided by perplexity (Section 2). Second, expecting some disagreement does not tell how large it is. The loss orders the rules very reliably (0.996), and that is all a reader of a loss table sees. Yet of the rule pairs that both a loss and a benchmark order with confidence, 15 to 44% are ordered in opposite directions, and the rule the WikiText-103 loss ranks first is on average 2.9 to 12.0 percentage points less accurate than the rule that is best on each benchmark (Section 5).

We did not mean the paper as an argument against held-out loss, and the submission did not make this clear enough. What we want it to say is smaller: an ordering given by the loss should be checked against the task before it is used to choose.

W3. Context length. You are right that 128 tokens is far from practice, and that the 512-token result was mentioned only briefly; it is now in Section 4, with half and with one eighth kept. We also ran the 21 rules at 3840 tokens on two question-answering tasks that we built in the style of RULER (Appendix C.5). Each has 400 questions, and each question comes with a context of exactly 3840 tokens: the paragraphs that answer it, set among distractor paragraphs that do not. In the first task this is one SQuAD paragraph among other SQuAD paragraphs. In the second it is the two paragraphs of a HotpotQA question, which is written to need both, among more distractor paragraphs than HotpotQA itself provides (yes/no questions removed). As on our SQuAD ruler, a question counts as answered when every token of the answer is the model's top prediction. The loss is read on PG-19 (books) and FineWeb (web pages), 512 windows each. WikiText-103 is not used there, because its test split is too short for 512 windows of 3840 + 256 tokens.

3840 tokensLoss and taskThe two tasksReversed (SQuAD-paragraph task, HotpotQA task), of the pairs both order with confidence
Three small models, half kept+0.67+0.8511%, 13%
Three small models, one eighth kept+0.73+0.694%, 4%
Qwen3-8B, half kept+0.38+0.8323%, 28%
Qwen3-8B, one eighth kept+0.62+0.6918%, 16%

The three small models are Qwen3-0.6B, Llama-3.2-1B and OLMo-2-1B. A task is used on a model only if, with the full context, the model answers at least 25% of its questions and at least 10 points more than the weakest of three simple reference rules, so that the rules have room to differ. Llama-3.2-1B answers 24% of the SQuAD-paragraph task, so the value between the two tasks is the mean over Qwen3-0.6B and OLMo-2-1B.

With half or one eighth kept on Qwen3-8B, and with half kept on the three small models, the loss agrees with the two tasks less than the tasks agree with each other. It agrees as well only with one eighth kept on the small models, where few questions are answered under any rule: even the best of the 21 rules answers only 7.5 to 13.5% of the questions there, against 25.2 to 41.5% with the whole context. This setting, the one nearest the long contexts and small budgets of published methods, goes against our finding at 128 tokens, and the paper now says so (Section 4). There the tasks reproduce their own ordering at only 0.63 to 0.92 across halves of their questions, against 0.99 or more for the loss, and part of the match may come from that, since low reliability lowers the agreement between two tasks more than that between loss and task. The low hit rate may also contribute; we have tested neither.

Is it the length or the small fraction? At 128 tokens, keeping less does not remove the disagreement: with one quarter or one eighth kept, the loss agrees with a benchmark at +0.46 and +0.36 on average, against +0.58 and +0.48 between two benchmarks, and 22 to 38% of the confidently ordered pairs are reversed. The comparison you cite shows the same: WikiText-103 agrees with the two CBT subsets at +0.16 and +0.36 at 128 tokens with one eighth kept, against +0.71 and +0.67 at 512 tokens (Appendix C.3). So the narrowing comes with the longer context, or with what changes along with it, and not with the smaller fraction alone.

The paper now states its finding for 128 tokens, from one half down to one eighth kept. At 3840 tokens the same disagreement appears in three of the four settings, but not in all, so the paper makes no claim for long contexts in general (abstract, Section 8, Appendix C.5). A test that holds the tasks and the texts fixed across lengths is left for future work.

W4. The task side.

Rare tokens. You are right. Section 4 now says that the lead of the rare-token rules is a mean over the four benchmarks at 128 tokens with half kept, and that the rule that keeps rare tokens is usually first on CBT only. At 3840 tokens it leads on one model at one fraction, and position, attention-and-position or norm rules lead elsewhere.

Which task. We agree that the four are not one task, and that SQuAD sits apart. We do not ask the loss to predict a single task. Each benchmark is the criterion for its own use, so the paper reports reversals and cost per benchmark (Section 5), and the benchmarks' disagreement with each other is the yardstick: two benchmarks agree at +0.64 on average, a loss on ordinary text and a benchmark at +0.44. On every model the mean agreement between a loss and a benchmark is below the mean between two benchmarks (Figure 1), although single pairs, such as PG-19 or FineWeb with LAMBADA, can agree more.

LongBench and RULER. We did not run them as released, and we ran no released compression method. Section 3.3 now says why we do not use them as released: both score text that the model generates, so a score also depends on how the answer is decoded and matched to the reference. We instead read the answer tokens under teacher forcing, as the loss reads its windows, so every rule is scored on the same answer and two rules differ only in the context tokens they keep. The two question-answering tasks of W3 are built in RULER's style, and so is a needle-retrieval task we added. On it the loss also misranks the rules with half kept (+0.35 over all six models, with 34% of pairs reversed), but the task agrees just as little with the two question-answering tasks (+0.34), and a rule's hit rate there tracks how often it keeps every token of the requested value (Section 4, Appendix C.5).

W5. The text the loss is read on. You are right, and the submission gave no reason for resting its headline on WikiText-103, the text that agrees worst with the benchmarks. The revision no longer rests on WikiText-103 alone and gives PG-19 and FineWeb more weight: Section 4 reports the three texts side by side, the abstract gives the range of reversals over all three, and each of the three results you name has a PG-19 or FineWeb counterpart beside it (Sections 5 and 6). The submission already gave them for the reversal counts and the trained selector; the counterpart for the opening pair is new, and the PG-19 and FineWeb reversal counts now use WikiText-103's word-frequency table. You are also right that, with each text's own table, PG-19 and FineWeb agree with LAMBADA at +0.71 and +0.74 (Figure 1), more than two benchmarks agree on average (+0.64). Even there, 19% and 17% of the rule pairs that both order with confidence are reversed, and the rule each loss ranks first costs 10.2 (PG-19) and 5.6 (FineWeb) points of LAMBADA accuracy, against 9.8 under WikiText-103 (Appendix C.2). The disagreement remains under these better texts (W6).

W6. The repair. You are right: changing the text does more than the gate. Mean agreement with the benchmarks rises from +0.26 to +0.34 with the gate (K = 16) and to +0.57 with FineWeb (+0.56 with its own word-frequency table). Section 7 now reports the change of text as a second repair, measured in the same way as the gate. Neither removes the disagreement: under FineWeb, 23% of the rule pairs that both the loss and a benchmark order with confidence are still reversed, against 36% under WikiText-103 and 33% under the gate, and the rule its loss ranks first still costs 5.9 points of accuracy on average, against 7.4 and 7.0. Section 7 also shows that the gate helps only when its window is much smaller than the 64 tokens a rule keeps: with windows of 32 and 64 tokens it agrees with the benchmarks less than the plain loss does.

Our comment to all reviewers summarizes what the paper claims and what changed in the revision.

Reviewer_L7uLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

As an initial caveat: I am no LLM expert and I can only assess the paper from a high-level view as AI/ML researcher. To my understanding, the paper asks whether held-out loss is a sound way to compare rules for shortening a language model's context (deciding which tokens to keep). It tests 21 rules, each keeping half of the tokens. Each measurement on its own is consistent (rank correlation ~0.96), but loss and task benchmarks rank the rules differently (correlation ~0.44), and disagree over which rule comes first. The authors conclude that losses shouldn't be abandoned, but be checked against the task before it's used to pick a rule. The paper validates its claims across 6 frozen open-source language models, 3 loss datasets and 4 task benchmarks.

Critical strengths

While I know little about the actual field of LLM training and context handling, the paper addresses the critical point of efficient context reduction which needs to be aligned with the underlying task. I think the paper investigates the important interplay between used held-out loss rules and underlying tasks and makes that interplay accessible (interpretable) which is important to improve held-out loss rules and further understand how to reduce context sizes effectively.

Critical weaknesses

For me, the central weakness of this work is that unfortunately at important places it is hard to understand/read, most likely due to the stated use of AI in translating / formulating by the authors. I myself cannot assess the works novelty over existing state-of-the-art.

Questions for authors

As a suggestion for future submissions, I think what would have supported me most in understanding the paper would be an easier to assess introduction: The work seems to jump right into a very complicated topic and assumes quite some background. One or two paragraphs at the very beginning placing it into a broader context for non practitioners would have helped.

3 comments

Thank you for the review, and for saying exactly what would have helped.

Readability. You are right. The submission was hard to read, and its introduction assumed too much background. The text is our responsibility, and we have revised it. We have uploaded the revised PDF.

  • Introduction. We rewrote it. It now opens with two paragraphs of background for readers outside the subfield (quoted below).
  • Terms and figures. Terms such as "frozen decoder", "ruler", "loss ruler" and "task ruler" are explained where they first appear in the introduction, and the captions of Figures 1 to 3 say how to read each figure.
  • Explanations and wording. We added explanations the main text lacked (three of them moved from the appendix into Sections 3.1, 3.3 and 5), split long sentences and revised the wording throughout the paper.

The new opening reads:

A language model keeps a key and a value vector for every token of its context (the KV cache), so its memory and time grow with the length of the context. To cut this cost, a selection rule decides which tokens stay. It gives every context token a score, for example from its position, the attention it receives or how rare it is, and keeps the highest-scoring fraction.

Which rule is used depends on how rules are compared. A rule runs while the model performs a task, so the direct test is a benchmark: apply the rule, give the model a task with a known answer, and check its answer. A cheaper test needs no labels: apply the rule to ordinary text and measure how well the model still predicts the tokens that follow. This is the held-out loss (perplexity is its exponential), which is precise as well as cheap. [...]

What is new. Published methods report held-out loss beside task scores, and some design choices have been settled by the loss alone. It is known that, across models, a lower loss tends to go with better task results. We know of no earlier work that measures whether, for one model and one family of rules, held-out loss on ordinary text and the tasks put the rules in the same order. The closest checks we found (Appendix H) use a different loss, or only observe that rules with similar perplexity can differ on tasks. We measure the agreement, after first checking that each measurement reproduces its own ordering, which rules out noise as the explanation. With contexts of up to 128 tokens, the loss agrees with the benchmarks less, on average, than the benchmarks agree with each other. Where the loss and a benchmark order a pair of rules differently, the loss usually orders that pair with 95% confidence. The paper audits how held-out loss is used to choose among rules, and proposes no new method.

Our comment to all reviewers summarizes what the paper claims and what changed in the revision.

We would be grateful if you could point us to any passage that is still hard to follow, and we will revise it.

Reviewer_L7uLOpenReview ↗

Response to authors rebuttal

The authors address central concerns, improved readability of the paper and updated their claims. While I still find it hard to assess the actual impact / relevance of the work and the breath of the claim (reliability established within sampled material, not necessarily across independent documents) I would raise my rating to weak accept.

Thank you for reading the revision and for your follow-up.

On impact and relevance: some published methods made design choices using held-out loss on ordinary text (Section 2). In our study, choosing that way has a cost: the rule ranked first by the loss on Wikipedia articles is on average 2.9 to 12.0 percentage points less accurate than the best rule on each benchmark (Section 5). The finding concerns context selection, but the check the paper recommends applies wherever a cheap loss is used to choose among costly options, such as pretraining data chosen by perplexity or checkpoints chosen by validation loss (Section 9).

On the breadth of the claim: you are right that the reliability is measured within the sampled material. Across independent documents the loss is less stable: the losses on three different texts (Wikipedia articles, books and web pages) agree on the order of the rules at +0.83, against 0.99 or more within one text. But this does not explain the disagreement. Removing the effect of this instability with the standard correction from measurement theory raises the agreement between the loss and the benchmarks only from +0.44 to +0.49, while two measures of the same thing would then agree at nearly 1 (Appendix B.3). The disagreement also appears with each of the three texts and on each of the six models (Figure 1).

Comments on the paper

1 comment

To all reviewers: what the paper claims and what changed in the revision

We thank the four reviewers. We have uploaded a revised PDF and reply to each review separately.

What the paper claims

  • Published methods that compress a model's context have settled some design choices using held-out loss on ordinary text, and a rule chosen that way is then used on a task. We test whether that step is sound: do the loss and the tasks put the same rules in the same order? We apply 21 selection rules to six frozen decoders (pretrained models whose weights stay fixed). The loss is measured on three texts (Wikipedia articles, books and web pages), and accuracy on four benchmarks (LAMBADA, CBT-NE, CBT-CN and SQuAD).
  • Each of these seven measurements gives almost the same ordering of the rules on two halves of its data (split-half reliability 0.956 to 0.996), so noise cannot explain the disagreement below.
  • Even so, with contexts of up to 128 tokens the loss and the tasks order the rules differently. With half of the context kept, the loss and a benchmark have a mean rank correlation of +0.44, against +0.64 between two benchmarks. Of the rule pairs that both a loss and a benchmark order with confidence, 15 to 44% are ordered in opposite directions, and choosing a rule by the loss costs accuracy on every benchmark on average.
  • How well the loss agrees with the benchmarks depends on the text it is measured on. Neither a small selector trained to minimize the loss nor a loss restricted to tokens that need distant context removes the disagreement.
  • Our claim covers contexts of up to 128 tokens, with one half down to one eighth of the context kept; there, keeping less does not remove the disagreement. At 3840 tokens the loss agrees with two question-answering tasks less than the tasks agree with each other in three of four settings: Qwen3-8B with one half or one eighth kept, and three small decoders with one half kept. In the fourth, the small decoders with one eighth kept, it agrees as well, but there few questions are answered under any rule and the tasks are measured less reliably than the loss. Because the disagreement holds in three of the four settings but not in all, we make no claim for long contexts in general; a test that holds the tasks and texts fixed across lengths is left for future work.
  • We do not claim that loss is a poor metric: measured on a benchmark's own labeled answers, loss and accuracy order the rules almost alike (+0.86 to +0.97). Nor do we rank any published method. Our point is that an ordering given by the loss should be checked against the task before it is used to choose a rule. The paper gives the procedure: estimate the reliability of the loss, then that of the task, and only then read their agreement.

What changed in the revision

  • Writing (L7uL, xhjD, 2HVw). We rewrote the introduction. It opens with two paragraphs of background and defines its main terms at first use. Long sentences are split, most headings of the results sections state their findings, and numbers that a figure already shows are repeated less often. Three explanations moved from the appendix into the main text: what the rules keep, what each benchmark asks, and how many reversals chance alone would give (Sections 3.1, 3.3 and 5).

  • The claim (2HVw, xhjD). The abstract and the introduction now state that the claim concerns held-out loss measured on ordinary text and used to choose among rules, and Section 8 states its scope.

  • New experiments (2HVw, L98i, xhjD), all reported in Section 4 with details in the appendix:

    • smaller fractions kept at 128 tokens, now on all six decoders and all seven measurements; the submission swept them on two decoders for one loss and one benchmark only (Appendices C.3 and C.5);
    • contexts of 3840 tokens, with two question-answering tasks and a needle-retrieval task (Appendix C.5);
    • the loss measured on each benchmark's own text (Appendix C.4);
    • one word-frequency table for all three texts, for the three rules that read such a table (Appendix A.2).

    The 512-token result was in the submission but mentioned only briefly; it is now in Section 4 as well (Appendix C.3).

  • Repairs (2HVw). Section 7 now compares two repairs on the same measures: restricting the loss to tokens that need distant context, and measuring the loss on web pages instead of Wikipedia articles.

  • Explanations. We also expanded explanations throughout the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.