- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to-predict-or-not-to-predict (TPNP), which chooses per-layer key and value compression schemes under a global KV-cache budget. It combines optional cross-head prediction with low-rank innovation coding. To choose the optimal strategy the method uses a callibration data and saves additional weights P_i, W_z, b_z, as well as compressed residuals which are used during online inference to both store a compressed KV Cache and to extract it.
The method performs well on several quality benchmarks, particularly at high cache-saving ratios.
Critical strengths
The decision to account for the storage cost of predictor anchors is well motivated. The mathematical analysis is sound and well presented. The comparison with prediction-free coding helps isolate when prediction is useful, and the experiments cover multiple models, budgets, and calibration sets. The analysis Section is interesting. I especially appreciate Figure 2 and the distribution of layer compression as well as the KV-assymetry.
Critical weaknesses
W1. The inference trade-off is unmeasured. The paper reports theoretical token-dependent cache size, but not decode latency, throughput, or peak inference memory. Reconstructing cached heads adds work at every decoding step. The useful comparison is generated tokens per second at a fixed GPU memory budget, including any batch-size increase enabled by compression. This remains the main systems concern. See more in the Questions section below.
W2. The benefit of prediction over a strong prediction-free variant is concentrated only at high compression as can be seen in the table in Appendix C. At 10–50% saving, results are often nearly tied or mixed. For example, at 50% on Yi, the prediction-free variant has higher average accuracy (59.87 versus 59.02). The clearest prediction-specific gains occur at 70%, where absolute quality also degrades substantially relative to the original models. The paper should more clearly separate the benefit of cross-head prediction from that of adaptive low-rank compression and allocation, and characterize when the added prediction machinery is worthwhile.
W3. The main architectural analysis covers a narrow KV-head regime. Qwen2.5-7B and Yi-1.5-6B both have four KV heads of width 128 per layer. Thus, the main experiments and key–value asymmetry analysis use the same size of per-layer anchor-partition search space. The additional backbones provide useful quality results, but do not show how partition choices, gains, and preprocessing cost evolve with the number of KV heads. This matters because candidate enumeration grows exponentially in that number.
Questions for authors
Q1. The evaluation does not establish whether the method improves practical decoding throughput. A newly generated token can be encoded directly into the compact representation, but attending to cached tokens requires reconstructing their K/V representations. As Appendix H.2 shows, this adds work proportional to context length for each decoding step, including target prediction and low-rank decoding. Appendix H.3 states that the implementation lacks dedicated fused kernels and does not report end-to-end latency or throughput. This omission matters because the proposed benefit is memory capacity during inference. Saving cache space may permit a larger batch, but reconstruction could reduce tokens per second enough to erase that gain. The theoretical cache-footprint comparison cannot resolve this trade-off.
Here's a short back-of-the-envolpe calculation I did for the additional calculation during decode:
Consider one layer, one newly decoded token, and earlier tokens in the KV cache. Let be the number of query heads and the head dimension. For each component , let be the number of cached anchor heads, the number of reconstructed target heads, and the number of cached innovation coordinates. Thus .
Without compression, computing attention scores and the weighted value sum takes approximately FLOPs, counting a multiply-add as two FLOPs.
TPNP performs this attention computation after reconstructing the target heads. For each cached token, it forms a prediction from the anchors and decodes the innovation: Using dense matrix products, this adds approximately
=2N\sum_{c\in\{K,V\}} \left(A_cT_cd_h^2+T_cd_hr_c\right)$$ FLOPs per decoding step, plus additions and positional rotations. The first term is target prediction; the second is innovation decoding. The key predictor has a RoPE-constrained structure, so its prediction term can be cheaper than this dense estimate. For Qwen2.5-7B we have $H_q=28$ query heads, $H_{\mathrm{kv}}=4$ KV heads, and head dimension $d_h=128$. Assume, for illustration, that both K and V use prediction-free low-rank coding: $A_K=A_V=0$, $T_K=T_V=4$, and $r_K=r_V=154$. Counting a multiply-add as two FLOPs, ordinary attention costs $$F_{\mathrm{attention}} =4NH_qd_h =4N(28)(128) =14{,}336N.$$ Reconstructing both cached K and V components adds $$\Delta F_{\mathrm{TPNP}} =2N\sum_{c\in\{K,V\}}T_cd_hr_c =2N\bigl[2(4)(128)(154)\bigr] =315{,}392N.$$ about $23$ times as much arithmetic. I might be wrong in my calculation, but I would like to at least see/read such a theoretical discussion. A much better demonstration would be to see the actual decode latency and throughput across context lengths and batch sizes, peak inference memory, and the maximum sustainable batch size at a fixed GPU memory budget. A comparison of total generated tokens per second at equal memory use would be particularly informative. If optimized reconstruction is required for the method to be competitive, the paper should quantify that requirement in the paper body rather than in the appendix. Q2. Could the authors extend the TPNP-versus-prediction-free comparison to long-context tasks and report the gain at each saving ratio? This would show whether prediction improves the outcomes most relevant to the proposed cache use case, beyond the perplexity and accuracy averages in Appendix C. Q3: Could the authors report partition counts or distributions, preprocessing time, and peak preprocessing memory for backbones with more KV heads? If a broader scaling study is infeasible, please qualify the scalability and key–value asymmetry claims accordingly. Q4. TPNP is generally strong in the main evaluations (Tables 1 and 2), but its relative advantage is less consistent elsewhere. On Qwen2.5-7B at 70% cache saving, it ranks third on RULER (Table 6), despite strong results on the other reported benchmarks. On the additional backbones in Table 8, it remains competitive but is not consistently best across metrics and saving ratios. I appreciate the inclusion of these results, and would welcome a discussion of why its relative ranking changes across long-context benchmarks and model architectures. Q5. In the second paragraph of Appendix D -- please refer to table 1.