acceptodds
Under review as a conference paper at ICLR 2027

When the Window Cannot See the Question: Token-Denominated Observation Windows Are Not Language-Neutral

Abstract

KV-cache evictors of the SnapKV family decide what to keep by scoring the cache from the last w tokens of the prompt alone. When those tokens do not hold the question, the ranking is blind: it keeps the question's own keys and drops most of the gold passage's. A padded instruction, a JSON schema and a tokenizer each put the question outside the window, and the tokenizer is the lever an English-validated benchmark cannot see. On Qwen3-4B one line of instructions and the chat suffix leave 25 tokens after the question in English, 107 in Bengali and 167 in Telugu, so a research library's default w=64 leaves English untouched and costs Bengali and Telugu double digits. The fix is a check, not a tuned constant: ŵ = c + Q90, the trailing block plus a question-sized slack, both measured before any forward pass, says whether a shipped window is safe for a prompt family, returns the smallest that is, and says when none is. It removes the blind mode and restores the passage, +14 over the default on the full Telugu pool at a tenth more prefill time; for every public static tail we found, a fixed 256-token window does as well. Neither removes the eviction cost a fully visible question still pays. Because no window costs English anything at the shipped ratio, an English-validated benchmark cannot tell the two shipped constants apart; the window is one token-denominated constant among several, and the check generalises: measure the constant on the tokenizer before trusting an English validation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, token eviction, SnapKV, multilingual, tokenizer fertility, long-context retrieval QA, observation window, query visibility, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GNg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a multilingual failure mode in KV-cache eviction methods that rely on a fixed observation window over the last-w-tokens, such as SnapKV. The authors show that tokenization differences can place the user's question outside this window, preventing the eviction algorithm from using question-related attention signals to identify important cached keys. Experiments demonstrate that this effect can cause substantial accuracy degradation, particularly for languages whose instructions are tokenized inefficiently. To address this issue, the paper proposes AutoWindow, a lightweight configuration check that determines the observation-window size from the trailing instruction length and a percentile of question lengths. The proposed approach recovers the accuracy lost under the default window in the evaluated settings.

Critical strengths

The paper demonstrates that an observation-window configuration that works well for English may substantially degrade performance in other languages because the same instruction occupies significantly more tokens. For example, under the default 64-token window, Bengali and Telugu experience double-digit accuracy drops while English remains largely unaffected. This is an interesting finding because conventional English-only evaluations could easily miss this failure mode.

Critical weaknesses

The multilingual degradation studied in this paper requires several conditions to hold simultaneously. The system must use a KV-cache eviction method based on a trailing observation window, the relevant question must be followed by sufficiently long content to fall outside that window, and the eviction ratio must be sufficiently aggressive. However, the authors' survey shows that common single-turn RAG and structured-output configurations generally place the question near the end of the prompt.

Moreover, mainstream serving frameworks such as vLLM and SGLang do not enable the studied eviction mechanisms by default. I suggest evaluating the failure mode in at least one realistic application workflow using its original prompt construction and task evaluation, preferably with the relevant KV-cache eviction mechanism enabled. This would establish whether the observed tokenization-induced blindness translates into meaningful task-level degradation.

This paper provides a convincing analysis of a specific KV-cache eviction failure mode, but its broader practical significance for multilingual LLM deployment remains uncertain.

Questions for authors
  1. The paper measures substantial growth in trailing context length across agent turns, but does not directly evaluate task-level performance. Do the authors have any existing results using native agent workloads with the relevant KV eviction mechanism?
  2. How should the observation window account for the fact that the original user question may no longer represent the information most relevant to the current step?
2 comments
AuthorsOpenReview ↗

Response to Reviewer GNg1

Thank you for the review and for the precise statement of the conditions. We agree with that statement: the paper's §8 conjoins the same three conditions (a last-ww evictor, a tail past the window, a ratio of about 0.75 or more), and App. U records that vLLM and SGLang do not enable these evictors by default.

On practical significance. Our claim is a measurement claim, not a deployment-incidence claim. Two points matter for multilingual deployment:

  1. Single-turn RAG is safe because its tails are English and short. Our 22-template survey (selected before reading) finds 0–5 tokens in retrieval QA and at most 6 in structured-output APIs (App. T). These are English counts. The same one-line instruction is 1.15× English in characters on Telugu but 8.1× in tokens. A suite that localises its tail moves it: on OneRuler's own native-speaker prompt the block exceeds 64 in 24 of 26 languages (App. T). The survey says English defaults are safe. It cannot say localised ones are.
  2. The diagnostic gap is the contribution. At the shipped ratio no window costs English anything, so an English-validated benchmark cannot tell w=64w{=}64 from w=247w{=}247. The check needs no forward pass and says whether a deployment is exposed.

Realistic workflow, end to end. We ran the closest public cases we could score. OneRuler ran unmodified on its own localised template (six languages). It is blind on Hindi and Tamil, yet loses nothing (0 fixed / 0 broken in all eleven cells), because the answer is a short needle that survives a gold-poor ranking. We report this as a negative result: blindness is necessary, not sufficient. For LlamaIndex's shipped refine template, rendered verbatim around our items, the submission had n=100n{=}100 only (Bengali −8.0-8.0, p=.057p{=}.057; English −3.0-3.0, ns). For this discussion we ran an arm preregistered in August: the full Telugu pool (n=669n{=}669) goes from 57.7 to 49.5 at the default window, −8.2-8.2 points (23 fixed / 78 broken, 95% CI [−10.6,−5.4][-10.6,-5.4]). The full Bengali pool (n=113n{=}113) gives −6.2-6.2 ([−11.2,+1.4][-11.2,+1.4], not significant), and we claim nothing for it. So one shipped prompt construction, unmodified, carries a certified loss on Telugu. This is still our QA items inside their template, one refine step. We have not run an application with its native task evaluation, and cannot before the deadline; §8 names this as a limitation.

Q1. Native agent workloads? No. We measure the geometry on 300 public trajectories per scaffold (OpenHands passes 256 at turn 2, median 340; about 2,000 tokens by turn 5). We also have one void arm, which we report without claims: SWE-agent's static block as the tail of our English items, void by its preregistered baseline floor (App. T). Task-level agent results are future work. For such prompts the check returns step 4: no window clears a growing tail, so protect or reorder.

Q2. When the original question is no longer what matters. This is the right question, and App. T flags it ("the question at step kk is not the task at step 0"). The rule is defined against whatever span should steer retention at the current prefill. For an agent turn that is the current step's instruction or observation, not the turn-0 issue. The natural extension measures ckc_k from the end of that span at each turn, which needs the scaffold to mark the span. Without that marker, the honest answer is the rule's fourth branch. We have added one sentence to §8 making this explicit.

Thank you again. If this clarifies the scope of the claim and what the check is for, we would appreciate it if you would consider revisiting your score.

Thank you for the response. I acknowledge that the paper identifies an interesting failure mode. However, the studied eviction mechanisms are not enabled by default in mainstream serving systems, and meaningful task-level impact in native application workflows remains unestablished. This leaves me concerned that the evaluation focuses on a somewhat artificial scenario with limited practical relevance. While the analysis is informative, I remain unconvinced that the overall contribution is substantial enough for ICLR. I will therefore maintain my original rating.

Reviewer_QPUDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a weakness in KV-cache compression methods such as SnapKV. These methods use attention from the last w tokens of a prompt to decide which earlier KV-cache entries to keep. If a long instruction comes after the user's question, the last w tokens may not include the question. The authors show that this happens more easily in some languages because the tokenizer uses many more tokens for the same kind of instruction. With Qwen3-4B and w = 64, the window includes the question in their English setup, but not in their Bengali and Telugu setups. Compression then causes large accuracy drops in the latter two languages.

The authors test this problem by changing the window size, padding English instructions, adding JSON-schema text, and using different models and tokenizers. They also measure query visibility: how much of the question lies directly inside the window. Their proposed tool, AutoWindow, recommends a window size using ŵ = c + Q90, where c is the number of tokens after the question and Q90 is the 90th-percentile question length. It is a way to check and adjust an existing setting, not a new compression algorithm.

The experiments cover eight languages and several models, including Qwen3 up to 32B. On the full Bengali and Telugu evaluation sets, AutoWindow improves accuracy by about 13 and 14 percentage points, respectively, compared with the default window. However, Telugu accuracy remains 5.7 points below the uncompressed baseline. The authors also find that placing the question last or using a fixed 256-token window often works as well in the static prompts they tested.

Critical strengths

1. Good experiments for identifying the problem

The authors do more than observe lower multilingual accuracy. Padding an English trailing instruction moves the failure condition without changing language, JSON-schema tails provide another non-language manipulation, and alternative tokenizers shift the relevant token counts. The visibility ladder gives a useful per-item predictor. The key-retention results also refine the story: a window that does not directly contain the question retains most question keys but keeps only about 41–43% of gold-passage keys, compared with 48–54% under a larger, question-visible window. The problem is not simply that the question itself gets evicted. These controls make the main empirical contribution credible.

2. Broad testing and clear reporting of negative results

The paper tests eight languages, several model sizes, different tokenizers, and multiple compression settings. It also reports cases where the proposed fix does not fully work and where the predicted effect is absent. The authors are open about these limits. This makes the paper useful as an empirical study, even though AutoWindow does not solve every compression problem.

Critical weaknesses

1. Limited evidence of practical benefits over simpler approaches

The paper does a good job of showing how tokenization can cause a fixed observation window to miss the question and reduce accuracy. However, I am less convinced that AutoWindow is necessary in most practical settings.

The authors show that moving the question to the end of the prompt or using a fixed 256-token window often produces similar results. They also demonstrate cases where AutoWindow performs better, such as a constructed Telugu prompt with a longer trailing instruction. However, it is unclear how often these situations occur in real applications.

I think the proposed method is useful as a way to check whether an observation window is large enough for a particular language and prompt structure. Still, the paper would be stronger if the authors demonstrated a real-world application where AutoWindow provides a clear advantage over simpler alternatives.

2. The explanation of question visibility needs clarification

The paper argues that when the observation window does not contain any question tokens, the eviction scores are independent of the question. I think this claim needs clarification.

In a multi-layer Transformer, tokens appearing after the question can already contain information about it through earlier attention layers. Therefore, even if the observation window does not directly include the question, its attention scores may still be influenced by the question.

This does not invalidate the experimental results. The authors provide convincing evidence that question placement and tokenization affect compression accuracy. However, I think it is important to distinguish between a window that does not directly contain the question and a window that cannot use any information about it.

An additional experiment measuring how the eviction scores change when only the question is modified would help clarify this point.

Questions for authors

Q1. Can an observation window that does not contain the question still use information from it?

The paper states that when the observation window does not contain any question tokens, the eviction scores are independent of the question.

However, in a multi-layer Transformer, tokens appearing after the question may already contain information about it through earlier attention layers.

Could the authors clarify this assumption? For example, would changing only the question, while keeping the passages and trailing instructions unchanged, affect the attention scores or the KV-cache entries selected for retention?

I think this would help distinguish between a window that does not directly contain the question and one that is completely independent of it.

Q2. In which practical scenarios does AutoWindow offer a clear advantage over simpler alternatives?

The paper shows that moving the question to the end of the prompt or using a fixed 256-token observation window often achieves similar results to AutoWindow.

The authors also provide cases where AutoWindow performs better, including a constructed Telugu prompt with a longer trailing instruction. However, I am unsure how frequently such situations occur in real-world applications.

Could the authors provide examples of realistic workloads where AutoWindow offers a measurable advantage over these alternatives?

It would also be useful to clarify the accuracy and computational trade-offs in such settings, particularly when changing the prompt structure is not possible.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QPUD (1/2): the question-swap test

Thank you for the close read, in particular for picking up the key-retention result (a blind window keeps the question's keys and loses the gold passage's). Both of your points lead to changes in the paper.

W2 / Q1. Can a window without the question still use information from it? Yes, and you are right that our sentence was too strong. SnapKV scores key jj by sj=∑t∈WAt,js_j=\sum_{t\in W}A_{t,j}, using layer-ℓ\ell queries of the window tokens. For ℓ>0\ell>0, those queries come from hidden states that have already attended to the question. So "no sjs_j depends on the question" is true, at best, of the first layer. The correct statement is: the window contains no question token, and the question can reach the ranking only indirectly, through the tail's residual stream. We have rewritten §3 this way (revised PDF).

We ran the test you suggested, with the design and the two possible readings written down before any data. Qwen3-4B, r=0.75r{=}0.75, 8k, 100 items per language. Only the question changes; passages and tail stay fixed. The donor is the item whose question has the nearest token length, which keeps positions aligned. Passage keys precede the question, so their cached entries are identical across the two runs and only the ranking can move. We compare the retained passage-key sets (Jaccard, mean over layers and heads; two random sets of these sizes would give 0.13) and the share of the gold passage's keys retained.

cellquestion in windowJaccard, own vs. swappedgold share, own → swapped
Telugu w=64w{=}64no0.9240.406 → 0.384
Telugu w^=247\hat w{=}247yes0.8400.481 → 0.418
Bengali w=64w{=}64no0.8920.428 → 0.373
Bengali w^=183\hat w{=}183yes0.8040.539 → 0.436
English w=64w{=}64yes0.8630.701 → 0.645

Your mechanism is real. With no question token in the window, swapping the question still changes 8% (Telugu) and 11% (Bengali) of the retained passage keys, and moves the gold share by +2.2+2.2 points ([+1.8,+2.6][+1.8,+2.6]) and +5.5+5.5 ([+5.0,+6.1][+5.0,+6.1]) toward the item's own question. By layer it behaves as you describe: the overlap is 0.99 at the first layer and falls to 0.82 (Telugu) and 0.75 (Bengali) by layer 24 of 36.

It is weaker than direct visibility. When the window holds the question, 16% and 20% of the keys change and the gold share moves by +6.3+6.3 and +10.3+10.3 points. The Jaccard difference between the two window types is +0.083+0.083 ([+0.077,+0.089][+0.077,+0.089]) for Telugu and +0.089+0.089 ([+0.081,+0.096][+0.081,+0.096]) for Bengali. Of our two registered readings, "blind in practice" and "question-informed through depth", the data fall in the second.

The swap also bounds a number already in the paper. At the shipped window the ranking keeps 41–43% of the gold passage's keys against 32% of a distractor's (§7). Under an unrelated question it still keeps 37–38%, so only 2 to 6 points of that contrast come from the question. What the indirect route does not do is close the gap: a window that holds the question keeps 48–54%, and across 19 cells the loss follows the gold share kept (Spearman +0.76+0.76).

The accurate summary is "question-absent window, question-informed but weaker ranking". The revision defines "blind" by the window's contents only, removes the independence sentence, and reports this table (§7, App. J). One caveat: the swap ran on a different stack (Apple MPS) from the paper's runs. It is prefill-only and compared within itself, and its gold shares at w=64w{=}64 (0.41, 0.43) match the paper's.

AuthorsOpenReview ↗

Response to Reviewer QPUD (2/2): when the rule beats simpler fixes

W1 / Q2. When does the rule beat question-last or a fixed 256? You put it accurately: it is mainly a check. Table 5 states that when the caller controls the layout, question-last is better (the full-pool Telugu residual is −3.4-3.4, against −5.7-5.7 at w^\hat w). It also states that a fixed 256 ties w^\hat w on Bengali and Telugu whenever the static tail fits. The rule wins only where both simpler fixes are unavailable:

  • A tail the caller does not author and that exceeds 256. Our constructed 270-token Telugu tail: w=256w{=}256 loses −20-20, w^\hat w loses −6-6, +14+14 head-to-head (p=.0013p{=}.0013). Public prompts of this kind exist. OneRuler's localised template leaves 264 (Hindi) and 345 (Tamil) tokens after the key, so a fixed 256 is blind there by geometry (that task happens to need little of the gold, so no loss; App. T). Agent scaffolds pass 256 by their second turn. There the check's answer is "no window is safe; protect or reorder".
  • Aggressive ratios. At r=0.9375r{=}0.9375 a 256-token window spends half of the ~512 retained tokens on itself and costs English −9-9; w^en=43\hat w_{\rm en}{=}43 costs −1-1 (App. O). The rule adapts downward, and a large constant cannot.
  • Cost. At r=0.75r{=}0.75, prefill goes from 1.07 s to 1.18 s at w^=247\hat w{=}247 (scorer share 6.5% → 15%). Peak memory and retained KV are identical (Table 8, App. B); 256 costs the same. So between 256 and w^\hat w, the trade-off is accuracy at aggressive ratios, not compute.

We do not have a deployed workload where the integer itself is decisive, and we say so. What we can show is that the measurement tells a deployer which of the three answers applies before running anything, and that an English validation cannot.

Thank you again. If the corrected claim and the swap test address your two points, we would be grateful if you would consider raising your score.

Reviewer_p32UOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how fixed token-based observation windows affect multilingual KV-cache compression. For SnapKV-style methods, trailing instructions can push the question outside the scoring window, with tokenization differences making some languages more exposed than others. The paper proposes AutoWindow, which sizes the window using the trailing instruction length and a held-out percentile of question lengths. Experiments show that enlarging the window can recover substantial accuracy losses at a fixed retained KV budget, although residual compression losses remain.

Critical strengths

The paper highlights a useful issue: a token-based configuration validated on English may behave differently across languages with different tokenization efficiencies. It demonstrates this issue concretely for observation-window-based KV eviction.

Critical weaknesses
  1. The practical relevance remains limited. The findings concern a particular class of lossy KV-eviction methods, rather than standard prefix-cache reuse in current serving systems (kv-cache use in SGlang/vLLM). As Appendix U acknowledges, these evictors are not the default paths in the surveyed production stacks. The paper also finds that many public QA templates avoid the problematic layout. While the mechanism study is useful, stronger evidence is needed to establish its impact on realistic deployed workloads.

  2. The long-context evaluation is limited in scope. Most experiments use QA examples packed with distractor passages into 8k-token prompts, with a supporting 16k experiment. This does not sufficiently establish whether the findings extend to substantially longer contexts or realistic long-document tasks, where KV compression is particularly relevant. The experiments on public templates and benchmarks provide limited additional evidence of accuracy degradation, and the agent analysis measures tail lengths without evaluating end-to-end task performance.

  3. The benefit and guarantees of AutoWindow are overstated. A fixed 256-token window performs comparably in the main Bengali and Telugu comparisons. Furthermore, the percentile-based rule does not cover every question, and even complete question visibility does not eliminate accuracy loss. The evidence supports a useful configuration heuristic, but not the stronger claim that it identifies the “smallest safe window.”

Questions for authors

Can the authors demonstrate the failure and recovery on realistic long-document workloads at substantially longer context lengths, comparing AutoWindow against a fixed larger window under matched KV budgets and reporting computational overhead?

1 comment
AuthorsOpenReview ↗

Response to Reviewer p32U

Thank you for the review. We agree with two of your three points in substance and have changed the paper; on the third we explain what the 16k slice and the mechanism already establish.

W1. Practical relevance. Agreed on the facts. This paper is not about prefix-cache reuse in vLLM/SGLang, which is lossless; it is about lossy last-ww eviction, which those stacks do not enable by default (App. U, cited against ourselves). The contribution is a measurement: an English-validated, token-denominated constant is not language-neutral, an English benchmark cannot detect it (no window costs English anything at the shipped ratio), and a check with no forward pass does detect it. The constant does ship as the kvpress default (w=64w{=}64) and in TensorRT-LLM's opt-in backend (32). Public templates mostly avoid the layout in English (App. T). On a localised public suite (OneRuler), however, the post-key block exceeds 64 in 24 of 26 languages. We report there that blindness costs nothing on a needle task, and we do not hide that result. New for this discussion: LlamaIndex's shipped refine template, rendered verbatim around our items, costs Telugu −8.2-8.2 points on the full pool (n=669n{=}669; 57.7 → 49.5; 95% CI [−10.6,−5.4][-10.6,-5.4]), in an arm preregistered in August (general response 2/2).

W2. Context length. The threshold is a property of the prompt's tail, not of the context: cc and Q90Q_{90} are measured before any forward pass and do not depend on prefill length. The 16k slice tests exactly that prediction. The default window loses −15.0-15.0 ([−19.7,−5.7][-19.7,-5.7]), and the same w^=247\hat w{=}247, without re-measurement, meets the gate (−2.0-2.0) and recovers +13.0+13.0 ([+4.6,+16.5][+4.6,+16.5]) (App. M). At a fixed ratio, longer contexts retain more keys in absolute terms but do not move the window–tail geometry. We do not claim magnitudes at 32k+ or on long-document tasks and cannot run them before the deadline; §8 states the 8k/16k scope. On public benchmarks: standard harness prompts leave 3–5 tokens after the question (App. T), so running them would measure nothing about this failure. That is why the paper holds tail length fixed by construction.

W3. "Smallest safe window" overstated. Agreed, and corrected. w^\hat w is the smallest window that covers 90% of held-out questions. By design it leaves the longest decile below V=1V{=}1 (§8), and a fully visible question still pays an eviction cost (per-item oracle −5.1-5.1; question-last −3.4-3.4). The revised abstract, contributions and §6 now say that w^\hat w is the smallest window covering 90% of the family's questions, not the "smallest safe" one. On the fixed 256: Table 5 states that it ties w^\hat w wherever the static tail fits, and the paper recommends it there. It fails past the tail (270-token tail: −20-20 against −6-6, +14+14 head-to-head) and over-protects at r=0.9375r{=}0.9375 (English −9-9 against −1-1).

Q. Longer realistic workloads vs. a fixed larger window, matched budgets, overhead. Matched budgets hold everywhere: SnapKV keeps the window inside the retained budget, so every comparison is at identical retained KV (2,041 keys per layer; peak memory 10.94 GB at 64, 247 and 256; Table 8). Overhead: prefill rises from 1.07 s to 1.18 s at w^=247\hat w{=}247 (+10%); 256 costs the same. Context: the 16k slice above, with the identical w^\hat w. A substantially longer, realistic long-document workload is not something we can run honestly within the discussion window, and we list it as the main open item.

Thank you again. Given the corrected claim and the scope clarification, we would be grateful if you would consider whether your assessment of soundness and relevance has changed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): scope, and the check vs. simpler fixes

We thank the three reviewers for careful reads. All three agree the failure is real and well identified (QPUD: "these controls make the main empirical contribution credible"; GNg1: "a convincing analysis of a specific KV-cache eviction failure mode"). The shared concerns are practical scope and whether the rule beats simpler fixes. We answer those here. Part 2/2 reports two new measurements run for this discussion and lists the PDF changes.

1. Practical scope: we agree, and the paper says it. The studied evictors are not on the default path of vLLM or SGLang (App. U; prefix-cache reuse is lossless and untouched by this paper). The exposure needs three conditions at once: a last-ww evictor, a tail after the question longer than the window, and a ratio of about 0.75 or more (§8). We do not claim deployed systems are failing today. The claim is narrower: a token-denominated constant validated in English is not language-neutral, an English benchmark cannot detect it (no window costs English anything at the shipped ratio), and two pre-forward-pass measurements say whether a given deployment is exposed. That diagnosis is cheap, and the same constants do ship: the kvpress research default is w=64w{=}64, and TensorRT-LLM's opt-in last-ww backend defaults to 32 (App. U).

Where the geometry exists outside our packing: (a) on OneRuler's own localised prompt (RULER in 26 languages), the post-key block exceeds 64 tokens in 24 of 26 languages, and Hindi (264) and Tamil (345) exceed even 256 (App. T). That task loses nothing (0 fixed / 0 broken in all eleven cells), because a four-token needle survives a gold-poor ranking. This is the "necessary, not sufficient" result the paper reports. (b) Iterated prompts: LlamaIndex refine leaves exactly 64 tokens, SWE-agent 221, and OpenHands passes 256 at its second turn (App. T). The refine template is now measured on the full Telugu pool and loses 8.2 points (part 2/2). End-to-end agent task performance is not measured; we say so in §8 and do not claim it.

2. The check vs. question-last and a fixed 256. Table 5 is the decision table, not a contest. If you control the layout, put the question last: V=1V{=}1 by construction and a smaller residual (−3.4-3.4 on the full Telugu pool). If the tail is static and fits in 256 tokens, a fixed w=256w{=}256 ties w^\hat w at every ratio we ran. The rule separates only where those fail: a 270-token tail (w=256w{=}256: −20-20; w^\hat w: −6-6; +14+14 head-to-head, p=.0013p{=}.0013), and aggressive ratios, where 256 over-protects (English −9-9 against −1-1 at r=0.9375r{=}0.9375). In both reviewers' phrasing, the value is the check — "is my window safe for this prompt family?" — more than the integer it returns.

AuthorsOpenReview ↗

General response (2/2): two new measurements, and what changes in the PDF

Both measurements were preregistered before any data existed, and both are reported whichever way they came out.

1. A shipped template on the full pool (GNg1, p32U). LlamaIndex's refine template, rendered verbatim around our items, puts the question about 8k tokens before the prompt end, so no item's question is inside the default window. The submission had this at n=100n{=}100 (English −3.0-3.0, ns; Bengali −8.0-8.0, p=.057p{=}.057) and called it uncertified. We wrote the full-pool arm in August and reserved it for this discussion. Qwen3-4B, r=0.75r{=}0.75, w=64w{=}64, against the layout's own uncompressed baseline:

nbaselinew=64w{=}64Δ\Deltafixed / broken95% CI
Telugu66957.749.5−8.2-8.223 / 78[−10.6,−5.4][-10.6,-5.4]
Bengali11376.169.9−6.2-6.24 / 11[−11.2,+1.4][-11.2,+1.4]

Telugu meets the registered "certified damage" reading (interval excludes zero at −5-5 or worse). Bengali points the same way and is not significant (p=.12p{=}.12); we claim nothing for it. Scope: our QA items in their template, one refine step, a deterministic scorer. It is not a LlamaIndex application with its own task evaluation, and blindness is still not sufficient (English in this layout and OneRuler lose nothing).

2. Question-swap test (QPUD Q1). We re-ran each prefill with only the question replaced by another item's question of the nearest token length (n=100n{=}100 per language). Passages and tail stay fixed, and passage keys precede the question, so only the ranking can change. When the window holds no question token, 8% (Telugu) and 11% (Bengali) of the retained passage keys still change, and the gold passage's share moves by +2.2+2.2 and +5.5+5.5 points toward the item's own question. When the window holds the question, 16% and 20% change and the share moves by +6.3+6.3 and +10.3+10.3. So Reviewer QPUD is right: a question-absent window is question-informed through depth, at roughly a third to a half of the strength. The table is in our reply to QPUD.

3. Wording we are correcting (revised PDF).

  • "returns the smallest [safe window]" → "the smallest window that covers 90% of held-out questions". Q90Q_{90} by design leaves the longest decile below V=1V{=}1 (§8), and full visibility still pays an eviction cost (oracle −5.1-5.1). p32U is right that "smallest safe" overstates this.
  • "no sjs_j depends on the question" (§3) → the window contains no question token. "Blind" is now defined by the window's contents only, and the swap numbers are reported (§7, App. J).
  • App. T and §8: the refine row changes from "benign on our items at n=100n{=}100" to the full-pool result above.

4. What is unchanged. No headline number moves. The full-pool results (Telugu +14+14, Bengali +13+13 over the default; the Telugu residual −5.7-5.7, [−7.8,−3.1][-7.8,-3.1]) are as submitted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.