acceptodds
Under review as a conference paper at ICLR 2027

SynKV: Rethinking Attention Structure in LLMs for Long-Context Code Generation

Abstract

As LLM-based code generation scales to repository-level inference, context lengths routinely exceed 100K tokens, sharply intensifying GPU memory pressure. Existing KV-cache compression methods remain limited in this setting: token-level selection struggles under aggressive KV budgets, while fixed-size grouping can break coherent code structures and substantially degrade generation performance. Program syntax provides a structural perspective, organizing code into locally coherent blocks. This raises a natural question: **does code-LLM attention mirror this block organization for KV retention?** Query-key interactions reveal syntax-aligned blocks whose recurring properties provide actionable selection signals. Motivated by this finding, we develop SynKV, a syntax-aware block-level framework for KV retention in long-context code generation that alleviates inference memory pressure while preserving generation quality. SynKV restructures flat token sequences into syntax-aligned blocks, then combines hierarchical dependency retrieval with lightweight summary-based relevance estimation to retain critical KV entries. Across four diverse code-generation benchmarks and six complementary metrics, SynKV advances the frontier in both memory scalability and generation quality. Notably, SynKV exhibits pronounced advantages under extreme long-context inference: (i) at the 1M-token scale, SynKV cuts the device-memory footprint from TB scale to tens of GB with only 0.005% KV retention, while existing systems run out of memory; and (ii) SynKV consistently sustains generation quality under aggressive compression, outperforming the best-performing method by over 40%. Beyond SynKV, our findings open a new direction in structured-attention design for scaling long-context LLM inference, balancing memory efficiency and generation quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Code Generation, KV-Cache Compression, KV Retention, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dohrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SynKV studies how to reduce KV cache memory for code generation with long contexts while keeping useful code information. It groups tokens into blocks based on program syntax, retrieves candidate blocks through scope and symbol dependencies, and uses block prefixes to estimate relevance. Selected blocks are reused within the current syntax block to reduce selection work. The paper also reports attention patterns that motivate these choices. Experiments cover four code benchmarks, two main models, and several cache budgets, with further model checks and component ablations. The authors report higher generation scores and lower total latency than the compared compression methods.

Critical strengths

1.Using code structure makes sense here, and the ablations support it. Section 5.4 reports about a 45% average drop on EB/RB when retrieval uses nearby blocks. Fixed chunks also hurt performance, supporting the choice of syntax blocks.

2.At the stated 10% budget and 128K context, both models exceed 40% RepoBench exact match and 50% CodeBLEU. Baselines score in single digits or run out of memory (Tables 1–2). This is a useful improvement.

3.The prefix summaries look like a useful way to save work. Section 5.4 reports that scoring whole blocks improves the average score by less than 2%, suggesting that the cheaper summaries keep most of the useful information.

Critical weaknesses

1.Eq. (5) does not explain how the reported cache budgets are met. If retention counts token positions, 0.005% at 1M means about 50, below Section 4.2’s 128-token minimum. The memory claim therefore needs clarification.

2.Section 5.2 describes SnapKV as using 128-token chunks, unlike its original token selection after pooling. The EvoBench reference on page 14 concerns text detection. These may be writing errors, but the actual evaluation setup needs clarification.

3.Table 3 does not state the model, budget, or output length. SynKV’s first-token latency is higher than StreamingLLM’s, so it remains unclear whether short completions would benefit from the reported total latency savings.

Questions for authors

1.Could you explain how Eq. (5) meets each budget and give the 1M run’s retained KV size and peak GPU memory? For Eq. (4), are summary lengths fixed, and is using total attention intentional?

2.Could you identify the EvoBench dataset and evaluation script, and confirm whether SnapKV follows its original selection rule? These details would help me check the comparisons in Sections 5.1–5.2.

3.Could you give Table 3’s model, dataset, budget, batch size, and output length, including whether parsing time is counted? Timings for short and long completions would help show when faster decoding makes up for the slower first token.

Reviewer_u3iLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors give an interesting observation that the attention patterns strongly correlate with syntax patterns in coding langugage models. Using this observation, they propose a new KV cache compression method: they retrieve relevant blocks using structural dependencies and estimate the importance of selected blocks via prefix summaries and attention scores from the model. Selected KV are reused across tokens with the same syntactic block which reduces selection overhead. The method is evaluated on Qwen 2.5 Coder 7b and DS Coder V2 Lite, with substantial improvements across them compared to baselines and with ability to use contexts of 1M tokens on these models.

Critical strengths

Proposed method looks like a great augmentation of current KV optimization strategies for coding LLMs. If applied to modern coding LLM architectures, such as DS V4 Flash or even DS V4.1 Flash, the KV cache might be essentially free, making long context generation even cheaper. Additionally, the observation about the correlation of the attention pattern and syntactic blocks is quite interesting and novel.

Critical weaknesses

Selected models -- DS V2 Lite and Qwen 2.5 7b -- do not support 1M context in their default setups, and the paper does not mention how they are extended to these contexts.

Additionally, the paper claims KV usage of 0.005% of tokens for 1M context, with block sizes 128-1k. 1 000 000 * 0.00005 = 50 tokens, which is less then one block, so this claim does not really make any sense.

The observation about the KV cache locality is interesting, but would benefit from additional validation. I would be more satisfied if some non-structural baselines are ablated, maybe this locality will also be visible in general/random text.

Even though the performance retention on benchmarks is impressive, absolute performance values are quite low. For LongCodeBench for example the reported patch validity is at 3%, which is much more than the baselines, but still makes the models mostly output garbage. Thus, even with KV size reduction this approach might not produce any practical benefit.

Also, SynKV greatly increases the TTFT. Compared to StreamingLLM with 23.3 seconds prefill for 256k tokens, SynKV reports 63.04 seconds for the same context. For agentic performance this might be passable, but for latency dependant workflows such as FIM this might be crucual and limit the usefullness of the method.

Questions for authors

How is the 1M token context implemented on the baseline models?

How is the 0.005% of used context claim is calculated?

Will the observation hold on other structured and unstructured text, such as poems, tables, images and random text?

Reviewer_cQZDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SynKV proposes syntax-aware KV-cache selection for long-context code generation. It organizes code into syntax-aligned blocks, retrieves candidates through scope and symbol dependencies, and estimates relevance using compact block summaries. Selected blocks are reused within the current syntax block to reduce selection overhead. The paper reports improved generation quality and memory scalability across several code-generation benchmarks.

Critical strengths
  1. Using program structure as a prior for cache selection is well motivated. Syntax-aligned blocks offer meaningful retention units for preserving related code and dependencies that token-level selection may fragment.
  2. The framework combines structural candidate retrieval, lightweight relevance estimation, and amortized selection. This addresses both identifying relevant information and reducing the computational cost of finding it.
  3. The evaluation considers several code-generation scenarios and complementary quality measures, including dependency awareness and functional correctness. Additional backbone experiments and component ablations broaden the investigation beyond a single setting.
Critical weaknesses
  1. Structural retrieval may exclude important dependencies before attention-based selection can assess them. Scope and symbol relations are imperfect proxies for semantic relevance, particularly with incomplete code, dynamic references, and cross-file interactions. Reusing the selection throughout a block may also miss dependencies introduced later.
  2. The operational meaning of KV retention and the memory budget is unclear. Dynamic retrieval requires an explanation of where previously unselected KV states reside and whether they can be recovered. Without distinguishing permanent eviction, offloading, recomputation, and sparse access, the reported compression ratios are difficult to interpret.
  3. Important evaluation adaptations are insufficiently specified. Some experiments extend beyond the stated native context support, while baseline architecture adaptations and conversion of tasks into a common completion format can materially affect results. These choices make the comparisons difficult to reproduce and assess.
Questions for authors

What fraction of genuinely relevant blocks is excluded by candidate retrieval, and how robust is selection when dependencies become apparent only after generation enters a block?

Where are unselected KV states stored, what costs are incurred when they are needed again, and how is the total memory budget enforced across all storage and temporary computation?

How are positional handling, context extension, and baseline adaptations implemented, and do the conclusions hold under validated configurations and the benchmarks’ original task formats?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.