acceptodds
Under review as a conference paper at ICLR 2027

HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

Abstract

We introduce **HySparse2**, a hybrid sparse attention architecture with **two-level KV sharing**. At the outer sharing level, **KV Bridging** adopts a YOCO-style structure with a self-decoder and a cross-decoder, but bridges only full-attention layers. The self-decoder uses hybrid sliding-window attention, while the cross-decoder uses hybrid sparse attention. The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder. At the inner sharing level, HySparse2 retains HySparse's core **KV Reuse** design with two refinements. First, it replaces block-level sparsity with *token-level* sparsity to enhance long-horizon agentic capabilities. Second, it removes the separate SWA branch from sparse layers and instead forces a sliding window of recent tokens into the sparse selection. This two-level KV sharing mechanism enables HySparse2 to construct all KV caches from self-decoder hidden states. During inference, prefill can therefore exit early after the self-decoder, skipping all cross-decoder layers. After posttraining, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing inference cost and KV-cache size.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM.+Agent.+Sparse attention.+KV cache sharing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KamsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

HySparse2 adapts HySparse to long-context agentic workloads through two levels of KV reuse. At the outer level, a YOCO-style architecture divides the model into a first-half self-decoder and a second-half cross-decoder. KV Bridging applies layer-specific K/V projections to the input hidden states of the self-decoder’s sole FA layer, producing distinct caches for four cross-decoder FA layers. At the inner level, sparse layers reuse the preceding FA layer’s KV cache and selection indices, as in HySparse. HySparse2 replaces selection in 64-token blocks with selection of 1,024 individual global tokens. It also forces the 128 most recent tokens into the same sparse-attention computation instead of maintaining a separate SWA branch. Prompt KV-cache construction can therefore stop after the first 25 layers, while generation still runs the full model. HySparse2 reports higher long-context retrieval and agentic-task scores than HySparse and Hybrid SWA. At 1M tokens, Figure 3 gives 2.92× and 5.02× fewer prefill FLOPs, respectively, and an FP8 KV cache of 2.69 GB versus 6.72 GB and 12.09 GB.

Critical strengths

S1. FA-only KV Bridging and the forced local window enable prompt KV-cache early exit. The four cross-decoder FA layers obtain their K/V from self-decoder hidden states, and subsequent sparse layers reuse those caches for both global and recent tokens. Unlike a separate SWA branch, this arrangement does not require local K/V derived from cross-decoder hidden states. The self-decoder/cross-decoder split follows YOCO, but its integration with HySparse’s within-group KV Reuse is a meaningful architectural adaptation.

S2. The reported long-context gains are substantial at a large training scale. Relative to HySparse, the 80B-A3B HySparse2 model raises pretraining RULER from 84.89 to 90.77 and NoLiMa from 40.27 to 49.76. After the common post-training stage, its average gains over HySparse are 11.30 points on MRCR-v2 and 19.81 points on RULER-v2.

Critical weaknesses

W1. The reported KV-cache reduction is explained by KV width rather than KV Bridging. YOCO lets its cross-decoder layers share one projected global K/V cache. HySparse2 instead gives each cross-decoder FA layer its own K/V projections and a distinct cache. Including the self-decoder FA layer, HySparse2 retains five global caches, as HySparse does. KV Bridging changes when these caches can be constructed, enabling prefill early exit, but does not reduce their number relative to HySparse. In Figure 3’s FP8 comparison at 1M tokens, HySparse’s per-token cache width is 5×4×(192+128)=6,4005\times4\times(192+128)=6{,}400, whereas HySparse2’s is 5×1×(256+256)=2,5605\times1\times(256+256)=2{,}560. This 2.5× ratio matches the reported 6.72-to-2.69-GB difference. The paper should distinguish the storage benefit of the MQA configuration from the prefill benefit of KV Bridging.

W2. The prefill benefit is not measured. Figure 3 reports calculated FLOPs and cache size, not elapsed prefill time. The paper also does not quantify token-level selection and gather overhead, end-to-end latency, or decode throughput in a serving implementation. Without these measurements, the practical speed benefit for the proposed agentic workload remains unverified.

Questions for authors

Q1. Please decompose the 1M-token cache sizes in Figure 3 by cached FA layers, KV-head count, K/V dimensions, and any local cache. After fixing KV-head count and dimensions, does KV Bridging itself reduce cache storage relative to HySparse? If not, please attribute the reported cache reduction to the changed attention configuration rather than to KV Bridging.

Reviewer_QjtCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HySparse2 combines KV Bridging and KV Reuse to reduce long-context cache storage and prefill computation. Token-level selection with forced recent-token coverage enables early-exit cache construction. Large MoE experiments report improved retrieval and trajectory likelihood over HySparse and Hybrid SWA, alongside analytical reductions in KV storage and prefill FLOPs.

Critical strengths
  1. The architecture addresses layer dependencies that prevent early-exit cache construction, going beyond simply reducing attention interactions.
  2. Substantial 80B and 290B experiments show promising long-context gains, with useful ablations of selection, local coverage, and bridging.
Critical weaknesses
  1. Changing KV heads from four to one and head dimensions from 192/128 to 256/256 already yields a 2.5x per-layer KV-size ratio. Matched-configuration comparisons are needed to separate these savings from the contribution of two-level sharing.
  2. Efficiency evidence is mainly analytical FLOPs and cache sizes. Please measure end-to-end prefill and decoding latency and peak memory, including selection and scheduling overheads, under matched hardware and configurations.
  3. AgentPPL and LongPPL measure existing trajectory likelihood, not closed-loop task performance. Please report task success and runtime on an interactive workload, or narrow the claims about agentic performance.
Questions for authors

Please check the weaknesses above.

Reviewer_mLrgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HySparse2, an upgraded version of HySparse designed for long-context, multi-turn agentic inference via two-level KV sharing. At the macro level, the model adopts a YOCO-inspired self-decoder/cross-decoder split. Through "KV Bridging," each cross-decoder full-attention (FA) layer constructs its KV cache directly from self-decoder FA hidden states via layer-specific projections. At the micro level, the model preserves HySparse's sparse-layer KV reuse while introducing two key modifications: (1) switching from block-level to token-level selection, and (2) replacing the standalone sliding-window attention (SWA) branch with a forced local window within the sparse selection. Together, these designs decouple cross-decoder KV dependencies from its own computation, allowing prefill to exit early immediately after the self-decoder. Empirical evaluations on 80B-A3B MoE models show that HySparse2 matches general benchmark performance while outperforming HySparse on long-context evaluations (RULER, NoLiMa, and +11.30 MRCR-v2 / +19.81 RULER-v2 gains after 256k post-training). The paper also demonstrates significant prefill computational savings (up to 5.02× lower FLOPs at 1M tokens) and validates key architectural choices with comprehensive ablations up to the 290B-A8B scale.

Questions

  • How were the baselines extended from 32k to 256k (e.g., which RoPE scaling method was used)?
  • Section 4.1 states that HySparse/HySparse2 remain comparable to the full-attention baseline, but no full-attention results are reported.
Critical strengths
  1. Practical Problem Formulation: The paper tackles a well-motivated, realistic bottleneck in input-dominated, multi-turn agentic workloads, where prefill latency and cumulative KV-cache memory dominate serving costs.
  2. Substantial Scale and Rigorous Training Controls: Experiments are executed at production scale (80B-A3B base models, with the KV Bridging ablation scaled to 290B-A8B). Baselines and proposed variants share identical pretraining data and optimization schedules, ensuring a fair foundation for architectural comparisons.
Critical weaknesses
  1. Conflation of KV-Cache Savings with Standard MQA & Lack of Wall-Clock Profiling:

    • Theoretical Memory Gains via Head Dimensions: The reported 2.5×2.5\times KV-cache reduction (6.72 GB down to 2.69 GB) appears mostly driven by head configurations rather than the proposed architectural mechanisms. In Table 1, HySparse2 uses single-head MQA, whereas HySparse retains multi-head allocations. Because both models retain exactly 5 FA KV caches, this reduction stems from applying standard MQA head-count reductions that could equally benefit the baselines, rather than KV Bridging itself.
    • Absence of Real Hardware Benchmarks: All efficiency claims rely strictly on theoretical FLOP counts. The submission lacks empirical system profiling—such as Time-to-First-Token, end-to-end prefill/decode throughput, and kernel execution time. In particular, token-level top-kk selection introduces non-contiguous memory access and irregular gather operations.
  2. Component contribution analysis:

    • Relative to HySparse, HySparse2 alters multiple major variables simultaneously: MQA head dimensions, positional encodings (adopting NoPE in FA/sparse layers, whereas baseline positional encodings remain unspecified), decoder layer splits, selection granularity, local-window formatting, and KV Bridging. This makes it hard to understand which component contributes best to the overall improvement.
  3. Discrepancy Between Motivation and Evaluation Protocol:

    • Missing Downstream Agentic Task Metrics: While the paper repeatedly frames the contribution around "multi-turn agentic inference," it omits standard downstream task-completion benchmarks (e.g., SWE-bench). The only agent-aligned evaluation is an internal "AgentPPL," where differences across variants are marginal (e.g., $\sim$1.160 vs. 1.162 vs. 1.165 at 64k).
    • Length Mismatch Between Efficiency and Quality: Prefill FLOP savings and memory projections are heavily highlighted out to 1M tokens, yet empirical generation quality and retrieval accuracy are evaluated only up to 256k.
Questions for authors
  • How were the baselines extended from 32k to 256k (e.g., which RoPE scaling method was used)?
  • Section 4.1 states that HySparse/HySparse2 remain comparable to the full-attention baseline, but no full-attention results are reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.