- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a multilingual failure mode in KV-cache eviction methods that rely on a fixed observation window over the last-w-tokens, such as SnapKV. The authors show that tokenization differences can place the user's question outside this window, preventing the eviction algorithm from using question-related attention signals to identify important cached keys. Experiments demonstrate that this effect can cause substantial accuracy degradation, particularly for languages whose instructions are tokenized inefficiently. To address this issue, the paper proposes AutoWindow, a lightweight configuration check that determines the observation-window size from the trailing instruction length and a percentile of question lengths. The proposed approach recovers the accuracy lost under the default window in the evaluated settings.
Critical strengths
The paper demonstrates that an observation-window configuration that works well for English may substantially degrade performance in other languages because the same instruction occupies significantly more tokens. For example, under the default 64-token window, Bengali and Telugu experience double-digit accuracy drops while English remains largely unaffected. This is an interesting finding because conventional English-only evaluations could easily miss this failure mode.
Critical weaknesses
The multilingual degradation studied in this paper requires several conditions to hold simultaneously. The system must use a KV-cache eviction method based on a trailing observation window, the relevant question must be followed by sufficiently long content to fall outside that window, and the eviction ratio must be sufficiently aggressive. However, the authors' survey shows that common single-turn RAG and structured-output configurations generally place the question near the end of the prompt.
Moreover, mainstream serving frameworks such as vLLM and SGLang do not enable the studied eviction mechanisms by default. I suggest evaluating the failure mode in at least one realistic application workflow using its original prompt construction and task evaluation, preferably with the relevant KV-cache eviction mechanism enabled. This would establish whether the observed tokenization-induced blindness translates into meaningful task-level degradation.
This paper provides a convincing analysis of a specific KV-cache eviction failure mode, but its broader practical significance for multilingual LLM deployment remains uncertain.
Questions for authors
- The paper measures substantial growth in trailing context length across agent turns, but does not directly evaluate task-level performance. Do the authors have any existing results using native agent workloads with the relevant KV eviction mechanism?
- How should the observation window account for the fact that the original user question may no longer represent the information most relevant to the current step?
2 comments
Thank you for the response. I acknowledge that the paper identifies an interesting failure mode. However, the studied eviction mechanisms are not enabled by default in mainstream serving systems, and meaningful task-level impact in native application workflows remains unestablished. This leaves me concerned that the evaluation focuses on a somewhat artificial scenario with limited practical relevance. While the analysis is informative, I remain unconvinced that the overall contribution is substantial enough for ICLR. I will therefore maintain my original rating.
Response to Reviewer GNg1
Thank you for the review and for the precise statement of the conditions. We agree with that statement: the paper's §8 conjoins the same three conditions (a last-w evictor, a tail past the window, a ratio of about 0.75 or more), and App. U records that vLLM and SGLang do not enable these evictors by default.
On practical significance. Our claim is a measurement claim, not a deployment-incidence claim. Two points matter for multilingual deployment:
Realistic workflow, end to end. We ran the closest public cases we could score. OneRuler ran unmodified on its own localised template (six languages). It is blind on Hindi and Tamil, yet loses nothing (0 fixed / 0 broken in all eleven cells), because the answer is a short needle that survives a gold-poor ranking. We report this as a negative result: blindness is necessary, not sufficient. For LlamaIndex's shipped refine template, rendered verbatim around our items, the submission had n=100 only (Bengali −8.0, p=.057; English −3.0, ns). For this discussion we ran an arm preregistered in August: the full Telugu pool (n=669) goes from 57.7 to 49.5 at the default window, −8.2 points (23 fixed / 78 broken, 95% CI [−10.6,−5.4]). The full Bengali pool (n=113) gives −6.2 ([−11.2,+1.4], not significant), and we claim nothing for it. So one shipped prompt construction, unmodified, carries a certified loss on Telugu. This is still our QA items inside their template, one refine step. We have not run an application with its native task evaluation, and cannot before the deadline; §8 names this as a limitation.
Q1. Native agent workloads? No. We measure the geometry on 300 public trajectories per scaffold (OpenHands passes 256 at turn 2, median 340; about 2,000 tokens by turn 5). We also have one void arm, which we report without claims: SWE-agent's static block as the tail of our English items, void by its preregistered baseline floor (App. T). Task-level agent results are future work. For such prompts the check returns step 4: no window clears a growing tail, so protect or reorder.
Q2. When the original question is no longer what matters. This is the right question, and App. T flags it ("the question at step k is not the task at step 0"). The rule is defined against whatever span should steer retention at the current prefill. For an agent turn that is the current step's instruction or observation, not the turn-0 issue. The natural extension measures ck from the end of that span at each turn, which needs the scaffold to mark the span. Without that marker, the honest answer is the rule's fourth branch. We have added one sentence to §8 making this explicit.
Thank you again. If this clarifies the scope of the claim and what the check is for, we would appreciate it if you would consider revisiting your score.