- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes LayerTrace, a learning-based sparse attention method that predicts KV selection for future layers using source-layer token representations. It jointly trains token encoders and destination-specific scorers against frozen dense attention, while retaining the original KV cache. The paper evaluates quality on LongBench v2 and a RULER-derived diagnostic, reports whole-decoder latency across context lengths and batch sizes, and explores compatibility with hybrid-attention and MLA architectures.
Critical strengths
The idea of predicting destination-layer KV selection from earlier source layers is potentially useful, as it aims to reduce repeated selection overhead during decoding.
Critical weaknesses
1. Severely incomplete evaluation. The evaluation is insufficient to support the paper's claims. Qwen3-4B is evaluated only on LongBench v2, while DeepSeek-R1-Distill-Llama-8B is evaluated on merely 56 RULER-derived questions, covering 7 of the 13 standard tasks with only 8 samples per task instead of the standard 500. Surprisingly, Appendix B reports complete RULER results (6,500 samples) for the baselines but excludes LayerTrace itself.
2. Questionable efficiency results and unclear experimental presentation. The efficiency evaluation contains several unexpected results that are not adequately explained. In Table 5, speedups do not consistently improve with increasing context length; for example, DeepSeek at batch size 2 drops from 1.10× at 128K to 0.97× at 256K. More concerningly, Appendix Table 16 reports lower dense-decoder latency at 32K than at 16K for Qwen under batch size 1. The baseline implementations also differ substantially in optimization level, with several failing numerical consistency checks, making the latency comparisons difficult to interpret. Furthermore, Table 4 does not clearly explain how the reported KV selection percentages relate to the configured budgets, such as 0.2N, N/8, and 4K. The manuscript contains extensive experimental records, but the presentation is fragmented and the central evidence is difficult to follow. There are also unfinished editorial statements, such as the AI disclosure indicating that the submission-form statement "will be confirmed before submission", which should not appear in a finalized manuscript.
Questions for authors
1. Benchmark coverage. Why are Qwen3-4B and DeepSeek-R1-Distill-Llama-8B evaluated on different benchmarks? Can the authors provide complete LongBench v2 and standard 13-task RULER results for both models? In particular, why is LayerTrace absent from the full RULER evaluation in Appendix B, despite all major baselines being evaluated?
2. KV selection accounting. How exactly is KV (%) in Table 4 calculated? Why does Quest with a 0.2N budget report approximately 32% selected KV, and why does SparQ with an N/8 budget report substantially different percentages across the two models?
3. Efficiency scaling. What explains the non-monotonic speedups in Table 5, particularly the performance regression of DeepSeek at 256K? Why is Qwen's dense-decoder latency at 32K substantially lower than at 16K under the same batch size?