W1. Limited Algorithmic Novelty and Marginal Improvements
My primary concern is the limited algorithmic novelty of the proposed method.
The underlying quantization approach is largely based on existing techniques, including asymmetric K/V quantization, channel-wise key quantization, and importance-aware mixed-precision allocation. ForeKV does not introduce a fundamentally new quantization algorithm. Instead, its main contribution is an importance estimation strategy that determines which cached tokens should receive higher precision.
However, the proposed importance estimator is relatively simple. It combines the mean of early decoding queries with the global prefill query mean through a directional-consistency-based interpolation. Similar ideas of predicting future query relevance and allocating KV cache resources according to token importance have already been explored in prior work.
More importantly, the empirical improvement attributable to the proposed global calibration mechanism is very limited.
The improvements in future-importance estimation shown in Figure 1(b) are also relatively small compared with using early decoding queries alone.
Furthermore, on Qwen3-8B, ForeKV achieves a LongBench average score of 55.82, compared with 55.71 for KIVI under similar KV cache compression ratios.
These results raise an important question: Does the proposed global calibration mechanism provide sufficient additional value over a simple early-decoding-based importance estimator?
I do not find the current experimental evidence convincing enough to establish the significance of the main algorithmic contribution.
W2. Missing Comparisons with Recent Strong KV Cache Quantization Baselines
The experimental comparison does not adequately cover recent strong methods in KV cache quantization.
The primary baselines are KIVI, ZipCache, PolarQuant, and ChanMix. While these are relevant methods, several more recent and closely related approaches are missing, including:
• Kitty (MLSys 2026): An efficient low-bit KV cache quantization method with dynamic precision adjustment and a customized inference implementation.
• MixKVQ (ACL 2026): A query-aware mixed-precision KV cache quantization method, particularly relevant to the importance estimation strategy proposed in ForeKV.
• PM-KVQ (ICLR 2026): A progressive mixed-precision KV cache quantization approach targeting long-chain-of-thought generation.
• KVmix (AAAI 2026): A mixed-precision KV cache quantization approach based on layer importance.
Although these methods differ in their allocation granularity or implementation requirements, they are highly relevant to the claimed algorithmic or system-level contributions of ForeKV.
The omission of Kitty is particularly concerning because the paper's efficiency evaluation adopts the GSM8K prompt used in Kitty's latency benchmark, yet no direct comparison with Kitty is reported.
Without comparisons against these methods under appropriately matched memory budgets, model configurations, and evaluation settings, it is difficult to assess whether ForeKV provides meaningful advantages over recent state-of-the-art approaches.
W3. Insufficient Validation of Future Token Importance Prediction, Especially for Long Generation
Another major concern is the insufficient validation of the central assumption behind ForeKV.
The method uses only the first eight decoding queries to estimate future token importance. After the initial calibration, the query representation remains fixed and is reused to determine the precision allocation of subsequently generated keys.
However, attention patterns may evolve substantially during generation. Tokens that appear unimportant in the early decoding stage may become important much later, particularly when generation involves multiple stages of reasoning or shifts between different topics.
Once a cached key has been assigned low precision, ForeKV does not revisit its precision allocation. Therefore, errors in the initial importance estimation may persist throughout the generation process.
Although Figure 1(b) evaluates future-importance estimation, the analysis only considers decoding windows up to step 192. This is insufficient to establish whether the initial query representation remains reliable over thousands of generated tokens.
The paper includes a 4096-token generation benchmark for throughput evaluation, but this does not directly demonstrate that the method preserves generation quality or accurately predicts future key importance over long outputs.
I believe this is a major limitation because the reliability of future token importance prediction is the central motivation of the proposed method.
A more convincing evaluation should examine how prediction quality changes across substantially longer decoding windows, such as 512, 2048, 4096, and 8192 tokens. It should also evaluate the downstream accuracy impact of using a fixed initial importance estimator.
W4. Incomplete Analysis of Additional Computational Overhead and Attribution of Speedup
The proposed framework introduces additional operations beyond standard KV cache quantization, including collecting early decoding statistics, scoring cached keys using calibrated queries, selecting important keys, reorganizing tokens, and packing the mixed-precision cache.
Although the key-scoring computation avoids materializing the full attention matrix, it still introduces additional computation across KV heads and Transformer layers. The cost of scoring, selection, and cache reorganization may become more significant as the input context length increases.
Furthermore, delaying compression requires temporarily retaining full-precision KV states during the initial decoding steps, potentially affecting peak memory usage for long prompts and large batches.
The paper reports end-to-end throughput and provides limited measurements of the effect of the early-decoding observation window. However, it does not provide a sufficiently detailed latency breakdown of the additional operations introduced by ForeKV.
There is also a concern about the attribution of the reported throughput gains. According to Appendix F, some baselines reconstruct dense KV tensors before attention, whereas ForeKV uses a customized fused attention kernel that directly processes the compressed cache.
Therefore, the reported 3.1× throughput improvement may reflect differences in kernel implementation and memory access patterns, rather than advantages of the proposed importance estimation algorithm itself.
A more detailed computational overhead analysis and appropriate kernel-level comparisons are necessary to distinguish the algorithmic contribution from the systems contribution.
W5. Limited Evaluation on Modern Reasoning Models and Insufficient Separation of Long-Context and Long-Generation Settings
Although the paper includes several recent Qwen3 and Qwen3.5 backbones, the evaluation lacks dedicated experiments on modern reasoning-oriented models, such as Qwen3-4B-Thinking-2507, or larger reasoning-oriented models.
More importantly, I believe the evaluation should clearly distinguish between long-input and long-output settings, as they introduce different challenges.
For long input contexts, the main concerns are whether importance estimation remains reliable over large KV caches and whether the additional selection and compression operations introduce significant time or peak-memory overhead.
For long generation, the main concern is whether the fixed early-decoding query representation remains predictive as attention patterns evolve throughout generation.
Thinking models provide another important evaluation dimension and should not be treated as equivalent to long generation. Their attention behavior should be evaluated across different input lengths and generation workloads.
I encourage the authors to include more systematic experiments covering different input context lengths and output lengths, ideally on modern reasoning-oriented models.
W6. Limited Clarity and Informativeness of the Figures
Several figures do not effectively communicate the main technical ideas.
Figure 1(a) visualizes changes in token importance but does not provide a sufficiently intuitive explanation of the evolving attention patterns. Furthermore, Appendix A.1 indicates that this visualization is based on a single constructed prompt and one selected layer/query head, which limits its value as general evidence for the motivation.
Figure 3 attempts to illustrate the entire pipeline, including query calibration, importance scoring, token selection, cache reordering, and quantization kernels. However, the diagram is overly dense, making it difficult to understand the exact relationships between these operations.
The paper would benefit from clearer and more technically precise figures, particularly a step-by-step visualization of the importance estimation, precision allocation, and quantization procedures.
W1-Q1. Effectiveness of Global Prefill Calibration
Thank you for your careful review and for pinpointing the issue with the original ablation.
We extended the key-selection ablation in Table 2 to Qwen2.5-3B-Instruct and Qwen3-4B on LongBench-11 and RULER-4K. Both methods use the same precision budget, attention kernel, and compression delay (Td=8). We used 2,000 paired bootstrap resamples to estimate 95% confidence intervals for the differences in macro-average scores.
Adding the prefill mean improved the average score in all four settings. The 95% confidence intervals exclude zero on both benchmarks for Qwen2.5-3B and on LongBench-11 for Qwen3-4B. The smaller gain on RULER-4K for Qwen3-4B has an interval that spans zero.
W2-Q2. Query Updates during Long Generation
Thank you for pointing out that a fixed query estimate may miss shifts in attention during long generations. We agree and plan to adopt periodic query updates in a future version of ForeKV. When packing a 128-token buffer, ForeKV-Update recomputes the estimate from the eight most recent decoding queries in that buffer and uses it to assign precision to the new keys.
Oracle Gap. We compared fixed and periodically updated query estimates on the same BF16 generation traces from Qwen3-8B with thinking enabled, using a top-40% selection budget.
Periodic updates reduced the Oracle Gap across the three intervals, bringing the selection of new keys closer to the oracle.
GSM8K accuracy. We evaluated the same model on all 1,319 test questions. Both quantized variants used 40% K4, 60% K2, and V2.
ForeKV-Update's overall accuracy was closer to the BF16 baseline. It answered three fewer questions correctly than the fixed variant, a drop of 0.23 percentage points. One possibility is that the update makes attention distributions more similar to those of BF16, producing more similar behavior on some questions.