- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Kronecker Product Attention (KPA), which represents token-dependent QKV matrices as sums of Kronecker products of compact matrix factors. By coupling head and feature indices within each factor, KPA allows different attention heads to have more diverse feature directions than standard Tensor Product Attention (TPA) at a given number of factor terms. The paper derives exact factorized attention computations and a compatible rotary operator, and reports improved average zero-shot accuracy over several attention baselines across four model scales, together with 6–10× fewer KV-cache entries than MHA.
Critical strengths
-
KPA is a meaningful extension of TPA. By increasing the dimensions of the Kronecker factors, it provides a way to increase representational capacity while retaining the benefits of structured factorization. I am interested in the possibility of exploiting the Kronecker structure to reduce computation and compute attention scores efficiently, although some of these benefits may already be attributable to TPA itself.
-
The paper examines how different factor allocations affect performance under comparable KV cache budgets. Although the choice of factor dimensions still appears to require empirical tuning, the reported results suggest that KPA can remain competitive across different allocations, indicating certain robustness and practicality of the method.
Critical weaknesses
-
The practical inference efficiency of KPA is not sufficiently established. The paper reports substantial reductions in the number of stored KV cache entries, but this does not directly establish comparable reductions in actual GPU memory usage or improvements in end-to-end inference speed. The factorized representation introduces its own computation and memory-access patterns, and the practical outcome will depend on how these operations are implemented. In particular, the paper does not provide end-to-end prefill and decode latency or throughput measurements, nor an analysis of kernel fusion, intermediate tensor materialization, and opportunities for parallel execution or overlap. The paper itself acknowledges the need for efficient factored decoding kernels.
-
The empirical evidence does not yet convincingly establish the breadth of KPA's performance advantages. The experiments provide preliminary evidence that KPA can outperform TPA. However, all experiments are conducted using Llama-based models and evaluated on a relatively narrow set of downstream benchmarks. Specially, alhough the reported accuracy of MHA, MQA, TPA are imported from the TPA paper, the reported accuracy of MQA is nearly identicaly or even better than that of MHA. This raises questions about the reported results and makes me less convinced by the claim that KPA substantially outperforms MHA and other common baselines. More comprehensive evaluations, including validation perplexity and tests on more advanced models and longer sequences, would strengthen the empirical evidence.
Questions for authors
-
The paper only report KV savings of KPA compared to MHA. Can the authors also report the comparisons to MQA, GQA, and MLA? Additionally, can they provide a preliminary analysis of the potential benefits of kernel fusion and reduced memory traffic?
-
Can the authors evaluate KPA on more advanced model architectures and report validation perplexity? Evaluations on longer sequences would also help assess whether the method preserves language modeling performance.
-
Since reducing KV cache storage is one of the main motivations for KPA, is it necessary to apply KPA to Q? Ablations comparing these design choices would help clarify this point.
-
Can the authors further analyze how the Kronecker-separable RoPE design affects performance?