- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CarryKV, a training-free hierarchical retriever for sparse attention. It summarizes temporal branches using query-independent cell counts and key sums, estimates branch attention mass through a count-weighted log-sum-exp readout, and maintains these summaries in a persistent carry forest.
Critical strengths
-
A coherent and interpretable approach to branch valuation.
-
Meaningful improvements over the evaluated hierarchical baselines.
Critical weaknesses
- The experiments do not establish a compelling practical quality–efficiency operating point.
The improvements over hierarchical baselines coexist with substantial degradation relative to dense attention. On Qwen3-8B, CarryKV obtains RULER scores of 69.16, 51.76, and 27.41 at 32K, 64K, and 128K, compared with 88.28, 74.32, and 58.29 for Dense.
The empirical comparisons are also concentrated on HiP and InfiniteHiP. Relevant sparse-prefill alternatives such as FlexPrefill, XAttention, and FlashPrefill V2 should be considered when assessing practical competitiveness. Their published results cannot establish superiority under this paper’s settings, but their existence makes consistent, matched-quality or matched-latency comparisons important. Without budget sweeps and stronger baselines, it remains unclear whether CarryKV offers a favorable tradeoff rather than improving a restricted baseline configuration.
- The model coverage is too narrow to substantiate broad applicability to modern long-context inference.
All evaluated backbones belong to the Qwen3 Base family. Modern hybrid architectures provide a materially different setting: for example, Qwen3.5-9B interleaves three Gated DeltaNet layers with one full Gated Attention layer.
Questions for authors
-
Can CarryKV achieve quality close to Dense attention while providing measurable end-to-end prefill acceleration, with quality, TTFT, and peak memory evaluated under the same configuration?
-
Does CarryKV retain an advantage over InfiniteHiP at matched latency and over strong sparse-prefill alternatives such as FlexPrefill, XAttention, and FlashPrefill V2 using their supported accuracy-preserving mechanisms?
-
Can CarryKV provide additional acceleration without substantial quality degradation when applied to the full-attention layers of modern hybrid models such as Qwen3.5?
3 comments
Dear Reviewer,
A revised manuscript has been uploaded. We have made several changes to address your concerns:
- Method clarification (Sections 3.1–4.3): We clarified the relationship between HiP and CarryKV, explicitly defined temporal branches, query blocks, and joint attention mass, and distinguished summary cells from tree children.
- Revised Figures 1–2 and method description: Figure 1 contrasts HiP's center-key valuation with CarryKV's multi-cell scoring and illustrates a concrete pruning example. Figure 2, together with the revised method description, clarifies bottom-up summary construction, branch selection, and retrieval of original K/V.
- Algorithm specification (Appendix B.1): We expanded the notation and traversal details, including frontier sizes, pruning decisions, and the scope of the logarithmic routing bound.
- Additional experiments (Appendix D.5): We added full-model quality–latency evaluations and explicitly distinguished the GPU-oriented fixed-depth implementation from canonical logarithmic traversal.
We appreciate your detailed feedback, which helped us improve the clarity and technical presentation of the manuscript.
Thank you for the additional experiments addressing stronger baselines and hybrid architectures.
My main concern remains: the higher quality configuration provides no meaningful acceleration over Dense, while the faster configuration is both slower and less accurate than native FlexPrefill.
I am therefore keeping my original score.
Thank you for recognizing our branch-mass formulation and highlighting the need for stronger practical evidence. We agree that its usefulness requires full-model evaluation. We have added latency/support-matched comparisons and a hybrid-model evaluation to address your questions.
Q1. Dense quality and end-to-end acceleration
CarryKV’s main contribution concerns branch valuation within HiP-style pruning. To examine its practical potential, we developed a GPU-oriented fixed-depth variant.
This changes the traversal, not merely the kernel: coarse scanning costs O(N/S) per query block and quadratic work over prefill at fixed span S and query-block width. The canonical O(log N) guarantee therefore does not apply. InfiniteHiP discusses a related parallelism–complexity trade-off (Infinitehip Appendix A.1). We will separate these new deployment results from the original algorithm’s guarantees.
On Qwen3-8B-Base at 64K/YaRN ×2, BF16, batch 1, A800, we evaluate 3,328 paired RULER examples (256 × 13 tasks). System results are means over the same fixed 10 independent sequences. Coarse branches cover 64 or 32 tokens and refine into 16-token nodes; Temporal-4 denotes four internal summary cells.
The 64-token variant reduces TTFT by 24.56% (1.325× speedup) with a 5.11-point lower score. The 32-token variant scores 1.14 points higher with 0.304% higher TTFT, approximately matching Dense latency. These offer different trade-offs; neither improves both metrics over Dense.
All three report 30.519 GiB mean peak allocated memory. CarryKV retains full-resolution K/V and adds routing summaries, without raw-cache eviction, compression, or offloading. We claim no peak-memory reduction; more efficient KV management remains future work.
Q2. InfiniteHiP and stronger sparse-prefill alternatives
To distinguish retrieval efficiency from implementation cost, we report both near-matched-latency and matched-support comparisons.
Near-matched latency. CarryKV’s 32-token variant achieves 75.22% at 10.890 s, versus InfiniteHiP’s 69.21% at 11.019 s: +6.01 points with 1.17% lower TTFT. InfiniteHiP uses remote budget 4,096, 32 sink tokens, and a 256-token local window. This is not a matched-support comparison.
Native and support-matched settings. Both CarryKV variants use 2,371.15 mean effective attention tokens. The corresponding baseline results are:
At matched mean support, CarryKV’s 32-token variant scores 4.35 points above FlexPrefill and 17.48 points above XAttention, but is slower. Its 64-token variant scores 11.23 points above support-matched XAttention with 6.40% lower TTFT.
The gains are not solely due to retaining more tokens. However, native FlexPrefill/XAttention remain faster than our higher-quality point. The advantage is therefore operating-point dependent.
Each method retains its supported accuracy-preserving mechanisms. Equal mean support does not imply equal per-query counts or search work; TTFT includes routing overhead. FlashPrefill V2’s released Hopper-only kernel is incompatible with A800, so we do not mix cross-hardware timings.
Q3. Hybrid architectures
We extend the evaluation to Qwen3.5-0.8B at 32K and 64K, with 3,328 examples per length (256 × 13). CarryKV replaces only full-attention layers; linear-attention components remain native. “Native” denotes the unmodified hybrid model.
CarryKV scores 0.95 points above Native at 32K and 1.02 points above at 64K, but TTFT is 20.55% and 5.66% higher, respectively. Thus, these configurations do not accelerate Native.
At 64K, it scores 1.83 points above FlexPrefill with 18.57% lower TTFT, and 1.13 points above XAttention with 37.68% lower TTFT. These comparisons use each method’s evaluated settings rather than matched support.
This extends the evidence to a hybrid architecture, but we do not extrapolate from this small model to Qwen3.5-9B.
Overall, these experiments support specific practical benefits, not universal superiority. We will report both the gains and limitations, separately from the canonical algorithm’s theoretical guarantees.