- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses an important practical problem in long-context inference: full-attention KV caches can remain a substantial memory bottleneck even when much of a model uses linear attention. Targeting this bottleneck is relevant to realizing the memory benefits of hybrid architectures. The proposed approach operates through cache selection without retraining the backbone, and its combination of reconstruction responses and linear-layer activations offers a concrete attempt to adapt eviction to these models. The reported improvements in several evaluated settings provide preliminary evidence of the combined policy’s practical value. Evaluation across multiple architectures and task families offers useful observations about its behavior under different conditions. In addition, the learned-gate variant and efficiency measurements reflect attention to the computational overhead of making eviction decisions, an important consideration for practical deployment. This inclusion of quality and computational-cost evaluations helps readers assess the approach as an inference method.
Critical strengths
The paper addresses a practically relevant problem: the memory cost of full-attention KV caches in hybrid linear–attention models. Its focus on cache selection without retraining the backbone makes the approach relevant to long-context deployment. The reported improvements in several evaluated settings provide preliminary evidence of the combined policy’s practical value. The presentation separates the motivation, method, and evaluation, and distinguishes the main procedure from its learned-gate variant. Qualitative examples provide an accessible illustration of the intended token-selection behavior. The appendix also collects pseudocode, hyperparameter settings, and detailed benchmark results, making these supporting materials convenient to consult. In addition, the learned-gate variant and efficiency measurements demonstrate attention to the computational overhead of cache management, extending the evaluation beyond prediction quality to practical inference costs.
Critical weaknesses
The paper’s main weaknesses are its insufficiently substantiated hybrid-specific rationale and an evaluation that does not fully establish the proposed components’ added value. Figure 1 shows greater query-wise attention dispersion, but this does not directly justify MRA’s aggregation across queries. Similarly, low linear-layer output norms do not establish that the corresponding tokens require protection in full attention, since output magnitude depends on the current query, recurrent state, and projections. The claimed connection between these proxies and eviction-induced degradation therefore remains unclear. Although the paper includes some ablations, LATP lacks matched random or uniform protection controls, leaving open whether its gains reflect complementary memory or simply a generic retention safeguard. Complete MRA × LATP comparisons are also missing on Olmo and Kimi, and the absence of KVzip+ leaves competitiveness against a stronger reconstruction-based scoring baseline unresolved. The breadth of the reported benefits also requires qualification: approximately 70% of the average improvement on Olmo/RULER comes from MK3 alone, while SCBench performance drops from 54.01 to 50.42 at a 70% eviction rate, limiting the generality of the “near-lossless” claim. These limitations materially weaken confidence in both the proposed mechanistic explanation and the generality of the empirical advantage.
Questions for authors
The following six issues are the most consequential for the review. Item 5 is a verifiable reporting error; the others are substantive gaps in the reasoning or experimental evidence.
- LATP’s central premise remains unsubstantiated: linear-layer output norms do not establish memory importance. Section 4.3 protects tokens with low output norms, but these norms depend on the current query, recurrent state, and projections. They do not directly indicate which information requires compensation from full attention. Without comparisons against random protection, uniform retention, and structural-token protection, the gains could simply come from reserving additional token positions rather than identifying “complementary memory.”
- There is a logical gap between MRA’s motivation and its scoring mechanism. Figure 1 shows that attention distributions are more diffuse for individual queries, whereas MRA changes how each cached token’s responses are aggregated across queries. Higher row-wise entropy does not directly justify replacing column-wise maximum aggregation with a Top-K mean. The analysis neither establishes this connection nor controls for confounders such as model size and training differences.
- The mathematical specification does not fully define the actual cache operations, leaving budget fairness difficult to verify. Equations 7–8 and Algorithm 1 do not sufficiently specify candidate sets, integer budgets, or how additionally retained tokens are deduplicated and counted. The position set and the tuple set also cannot be directly combined by a set union as written. Moreover, the comparability of raw scores used for selection across layers and heads is unverified. These ambiguities affect how much cache is retained, where it is allocated, and whether competing methods use equivalent budgets. They establish an incomplete specification, not a demonstrated implementation bug.
- The experiments do not adequately attribute the gains to the proposed mechanisms. Although the paper includes some ablations, it lacks a complete MRA × LATP factorial comparison on Olmo and Kimi, as well as comparisons with important scoring baselines such as KVzip+. The Fast variant also changes both the scoring target and the protection mechanism, requiring separate comparisons under matched training conditions. The results support the effectiveness of the combined method, but do not sufficiently establish the necessity or contribution of each component.
- Table 1 contains directly verifiable category-reporting errors. Several Single-Doc QA and Multi-Doc QA entries are inconsistent with Table 6. For example, Qwen-HybriKV should report 50.95 for Single-Doc QA and 45.84 for Multi-Doc QA, but Table 1 reverses these values. Several Olmo rows exhibit the same issue. Because the reversal does not affect every row, swapping the column headers would not resolve the problem; the entries require individual verification against the original results.
- Key claims exceed what the results support. On Olmo/RULER, approximately 70% of the 8.03-point average improvement comes from a single subtask, MK3, which does not support the claim that the improvement is “not driven by a single category.” On SCBench, performance decreases from 54.01 to 50.42 at a 70% eviction rate, so “near-lossless” requires explicit qualification. These results remain useful, but do not establish consistently broad, near-lossless improvements.