acceptodds
Withdrawn from ICLR 2027

Exploring a Layer-Wise Design Space for KV Cache Eviction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression. Code is available at https://anonymous.4open.science/r/PolyKV-iclr/README.md.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache eviction, KV cache compression, long-context language models, layer-wise method routing, cache capacity allocation, heterogeneous KV cache management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZYtpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether different KV-cache eviction methods should be assigned to different Transformer layers. It introduces POLYKV, a framework for constructing and executing static layer-wise plans using offline representation-preservation or perplexity profiles. Controlled experiments separate method routing, capacity allocation, and placement effects. The results support placement-sensitive routing in the evaluated settings.

Critical strengths
  1. The experimental design separates several potentially confounded factors. The routing-by-allocation factorial distinguishes method choice from cache capacity, while count-matched random placements isolate the contribution of layer assignment beyond the method mixture.
  2. The placement analysis provides evidence beyond a single favorable configuration. The guided route ranks second among count-matched assignments, and the frozen profile correlates with downstream performance across random controls.
  3. Static routing requires neither model retraining nor online search. The common execution interface enables composition of existing methods.
Critical weaknesses
  1. The performance gains appear modest and uneven across tasks. The evidence supports favorable average behavior, but does not yet establish a consistently useful improvement across workloads.
  2. Some comparisons do not fully isolate the benefit of layer-wise routing from differences in when compression is applied. This makes it difficult to attribute the observed improvements specifically to heterogeneous method assignments.
  3. The profiling procedure may not accurately capture a method’s suitability within a heterogeneous configuration. Local preferences can change when neighboring layers use different methods, while independent layer selection overlooks these interactions.
Questions for authors

How robust are the gains across repeated evaluations and task families, and what explains the tasks on which routing hurts performance?

Does routing retain its advantage when all baselines use identical compression phases and cache-management schedules?

How sensitive are method preferences to the surrounding assignments, and would profiling conditioned on the current route improve selection?

Reviewer_pMitOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates layer-wise heterogeneous KV cache eviction, questioning the conventional practice of applying a single eviction method across all Transformer layers. The authors introduce PolyKV, a framework that assigns existing eviction methods and cache capacities to individual layers using offline profiling based on representation similarity or perplexity degradation. The paper separates method routing from capacity allocation and evaluates their effects through controlled comparisons. Experiments on LongBench and RULER show that profile-guided routing can outperform homogeneous policies under certain settings, with additional evidence that the placement of eviction methods across layers affects performance.

Critical strengths
  1. Important and meaningful research question. The paper explores whether eviction-method identity itself should vary across Transformer layers, rather than focusing solely on layer-wise cache capacity allocation. This is a relevant design dimension for KV cache compression, and the distinction between method routing and capacity allocation provides a useful perspective for studying heterogeneous cache management.
  2. Controlled evaluation of layer-wise placement. The count-matched placement experiment is particularly informative. By fixing the number of D2O and TOVA layers while varying their locations, the authors provide evidence that performance depends on where eviction methods are assigned, rather than only on their overall composition. The guided placement ranks second among 100 evaluated assignments, and the offline profile scores exhibit a positive correlation with downstream performance (Spearman ρ = 0.459). These results support the existence of exploitable structure in the layer-wise routing space.
Critical weaknesses
  1. Limited methodological novelty and justification of the routing heuristic. PolyKV primarily combines existing eviction methods using layer-wise rankings derived from offline profiling. However, the routing procedure selects methods independently for each layer without accounting for interactions between assignments. In particular, cosine profiles are measured by applying each method across all active layers, whereas the resulting route combines different methods. It remains unclear how reliably these profiles predict performance under heterogeneous routing. Furthermore, profiling uses only 20 short calibration sequences at a fixed capacity, leaving the robustness of the selected routes across different settings insufficiently explored.
  2. Modest performance improvements and limited evidence of generalization. The primary matched LongBench comparison shows a small improvement of +0.2638 points, with a 95% confidence interval of [+0.0005, +0.5976] that only narrowly excludes zero. The improvement is also uneven across tasks, with one task accounting for 53.6% of the total gain, while the allocation-related contrasts are not statistically significant. Although the budget sweep reports consistent improvements, the standalone baselines are not fully matched in prefill/decode scope. Moreover, the guided RULER route does not outperform standalone TOVA, and the system-level evaluation uses a different fixed composition. These results leave the generalizability and practical significance of the improvements insufficiently established.
Questions for authors
  1. Routing heuristic and profile robustness. How sensitive are the selected routes to the calibration dataset, calibration capacity, and context length? Could the authors compare the current profile-guided routing with alternative selection strategies, including layer-local cosine profiling or methods that account for interactions between layer assignments?
  2. Statistical significance and controlled comparisons. Could the authors provide additional validation of the relatively small LongBench improvements, particularly across independent evaluation samples or another model? Additionally, could the budget-sweep comparisons be repeated under strictly matched prefill/decode scopes to determine whether the reported advantages over standalone baselines persist?
  3. Practical efficiency and generalization. Could the authors evaluate the actual profile-guided route, rather than a separate H2O/TOVA composition, in terms of GPU memory, latency, and throughput under matched resource constraints? It would also be useful to clarify whether independently constructed routes consistently outperform strong homogeneous baselines across different models and workloads.
Reviewer_77wiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the mitigation of growing KV caches with long-context inference via KV cache eviction methods. Specifically, the paper studies this problem along two axes: method routing, which determines how to evict cache at each layer, and capacity allocation, which allocates token budget across layers for routing. Experiments across two models show modest improvements over the RULER benchmark.

Critical strengths
  1. The idea to add layer-wise dynamism in expanding the design space for KV cache eviction strategies is novel and interesting.

  2. While the chosen setups for the empirical analysis are limited in breadth and impact (see weaknesses), it does a good job in going deep into the utility of the algorithm within the selected setup.

Critical weaknesses
  1. Unclear if this is a practical method that will generalize at modern scale. Specifically, algorithmic decisions are closely tied to the calibration datasets. On the other hand, the limited evaluation on outdated models (eg, llama, qwen3) make it hard to assess the generalizability of the method.

  2. The motivation for the work is long-context inference. But, the contexts used are quite small by modern standards, going up to 32K based on what I could infer.

  3. Unclear if there are any benefits to capacity allocation based on the marginal/mixed empirical results.

Questions for authors
  1. How sensitive is the eviction policy to shifts between calibration datasets and real-world prompts? Specifically, it would be interesting to see performance for other tasks like code generation, multi-turn conversational benchmarks.

  2. What is the impact of the pretrained model on these decisions? Specifically, i am curious to see the effect of model architecture (MoE vs not) and context length (eg, beyond 128k to increasingly common standard of 1M) for some recent models eg, gpt-oss, qwen3.8, gemma4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.