- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SynKV studies how to reduce KV cache memory for code generation with long contexts while keeping useful code information. It groups tokens into blocks based on program syntax, retrieves candidate blocks through scope and symbol dependencies, and uses block prefixes to estimate relevance. Selected blocks are reused within the current syntax block to reduce selection work. The paper also reports attention patterns that motivate these choices. Experiments cover four code benchmarks, two main models, and several cache budgets, with further model checks and component ablations. The authors report higher generation scores and lower total latency than the compared compression methods.
Critical strengths
1.Using code structure makes sense here, and the ablations support it. Section 5.4 reports about a 45% average drop on EB/RB when retrieval uses nearby blocks. Fixed chunks also hurt performance, supporting the choice of syntax blocks.
2.At the stated 10% budget and 128K context, both models exceed 40% RepoBench exact match and 50% CodeBLEU. Baselines score in single digits or run out of memory (Tables 1–2). This is a useful improvement.
3.The prefix summaries look like a useful way to save work. Section 5.4 reports that scoring whole blocks improves the average score by less than 2%, suggesting that the cheaper summaries keep most of the useful information.
Critical weaknesses
1.Eq. (5) does not explain how the reported cache budgets are met. If retention counts token positions, 0.005% at 1M means about 50, below Section 4.2’s 128-token minimum. The memory claim therefore needs clarification.
2.Section 5.2 describes SnapKV as using 128-token chunks, unlike its original token selection after pooling. The EvoBench reference on page 14 concerns text detection. These may be writing errors, but the actual evaluation setup needs clarification.
3.Table 3 does not state the model, budget, or output length. SynKV’s first-token latency is higher than StreamingLLM’s, so it remains unclear whether short completions would benefit from the reported total latency savings.
Questions for authors
1.Could you explain how Eq. (5) meets each budget and give the 1M run’s retained KV size and peak GPU memory? For Eq. (4), are summary lengths fixed, and is using total attention intentional?
2.Could you identify the EvoBench dataset and evaluation script, and confirm whether SnapKV follows its original selection rule? These details would help me check the comparisons in Sections 5.1–5.2.
3.Could you give Table 3’s model, dataset, budget, batch size, and output length, including whether parsing time is counted? Timings for short and long completions would help show when faster decoding makes up for the slower first token.