acceptodds
Under review as a conference paper at ICLR 2027

LayerTrace: Learning Source-Specific Token Latents for Sparse Attention

Abstract

Long-context decoding repeatedly reads historical keys and values, while sparse attention adds work to identify useful entries. LayerTrace moves that decision to an earlier transformer layer and learns the token representation used to make it. Each source stores a latent for every historical token, obtained from its source-block output through trainable content and positional projections. Independent destination-layer and query-head scorers use the current source output to prepare future read maps. Encoders and scorers are trained together against dense attention distributions and outputs, with a frozen backbone; destinations attend to selected original keys and values. On 503 LongBench v2 questions, Qwen3-4B at threshold 2 answers 157 correctly versus Dense's 153 while selecting 31.58% of original KV. In separate fixed-input timing at the same operating point, 32K context and batch 16, whole-decoder time falls from 62.68 to 36.34 ms per step. A 56-question DeepSeek development diagnostic scores 59.08 versus 49.40, but all-selected numerical discrepancies remain unresolved. Higher thresholds can collapse quality, and single-request execution can remain slower. These exploratory observations do not yet establish quality-preserving speedups under matched batched answer generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, sparse attention, efficient inference, cross-layer prediction, learned token representations, KV-cache access, long-context decoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ku9GOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LayerTrace, a learning-based sparse attention method that predicts KV selection for future layers using source-layer token representations. It jointly trains token encoders and destination-specific scorers against frozen dense attention, while retaining the original KV cache. The paper evaluates quality on LongBench v2 and a RULER-derived diagnostic, reports whole-decoder latency across context lengths and batch sizes, and explores compatibility with hybrid-attention and MLA architectures.

Critical strengths

The idea of predicting destination-layer KV selection from earlier source layers is potentially useful, as it aims to reduce repeated selection overhead during decoding.

Critical weaknesses

1. Severely incomplete evaluation. The evaluation is insufficient to support the paper's claims. Qwen3-4B is evaluated only on LongBench v2, while DeepSeek-R1-Distill-Llama-8B is evaluated on merely 56 RULER-derived questions, covering 7 of the 13 standard tasks with only 8 samples per task instead of the standard 500. Surprisingly, Appendix B reports complete RULER results (6,500 samples) for the baselines but excludes LayerTrace itself.

2. Questionable efficiency results and unclear experimental presentation. The efficiency evaluation contains several unexpected results that are not adequately explained. In Table 5, speedups do not consistently improve with increasing context length; for example, DeepSeek at batch size 2 drops from 1.10× at 128K to 0.97× at 256K. More concerningly, Appendix Table 16 reports lower dense-decoder latency at 32K than at 16K for Qwen under batch size 1. The baseline implementations also differ substantially in optimization level, with several failing numerical consistency checks, making the latency comparisons difficult to interpret. Furthermore, Table 4 does not clearly explain how the reported KV selection percentages relate to the configured budgets, such as 0.2N, N/8, and 4K. The manuscript contains extensive experimental records, but the presentation is fragmented and the central evidence is difficult to follow. There are also unfinished editorial statements, such as the AI disclosure indicating that the submission-form statement "will be confirmed before submission", which should not appear in a finalized manuscript.

Questions for authors

1. Benchmark coverage. Why are Qwen3-4B and DeepSeek-R1-Distill-Llama-8B evaluated on different benchmarks? Can the authors provide complete LongBench v2 and standard 13-task RULER results for both models? In particular, why is LayerTrace absent from the full RULER evaluation in Appendix B, despite all major baselines being evaluated?

2. KV selection accounting. How exactly is KV (%) in Table 4 calculated? Why does Quest with a 0.2N budget report approximately 32% selected KV, and why does SparQ with an N/8 budget report substantially different percentages across the two models?

3. Efficiency scaling. What explains the non-monotonic speedups in Table 5, particularly the performance regression of DeepSeek at 256K? Why is Qwen's dense-decoder latency at 32K substantially lower than at 16K under the same batch size?

Reviewer_RW3oOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LayerTrace, a sparse attention method designed for long-context decoding. In this approach, a "dense" source layer is designated every pp layers. Each source layer stores a learned latent representation for every historical token and generates a "read bitmap" for the query heads of several subsequent "destination layers." These destination layers then retrieve the original KV (key-value) pairs based on the bitmap, eliminating the need for additional selection mechanisms. The backbone model remains frozen, while an encoder and a scorer are trained via distillation, using the dense attention distribution and attention output as targets.

Critical strengths

Moving the selection out of the destination layer’s critical path—and amortizing the indexing overhead across multiple layers and heads—is an intuitive and sound approach. Profiling confirms a significant reduction in the time spent on attention in the destination layer.

The experimental report is refreshingly candid: it includes a comprehensive threshold sweep, numerical consistency checks, and even reports negative outcomes, such as a slowdown when the batch size (B) is 1.

Critical weaknesses
  1. Lack of clarity regarding novelty; insufficient discussion of related work.
  1. The evaluation is insufficient to support the conclusions.
  • LongBench v2 consists of four-choice multiple-choice questions. The dense baseline achieves only 30.42%, which is close to random guessing (25%). The performance gap between LayerTrace and the dense baseline is a mere 4 out of 503 questions, and the paired confidence interval includes zero, indicating a lack of meaningful discriminative power.
  • The evaluation on the 56-question development diagnostic set is problematic: half of these questions were seen beforehand, and there are issues with the numerical verification.
  • While Appendix B.1 presents results for all baselines on the full 13-task RULER benchmark, the results for LayerTrace are conspicuously absent.
  • The comparison allows different methods to utilize different KV budgets, making it difficult to assess the true quality-efficiency tradeoff.
Questions for authors

In the "Related Work" and "Introduction" sections, incorporate a discussion of the aforementioned studies and clearly articulate the novelty of LayerTrace in relation to IndexCache and the DSA family of methods. If feasible, include a comparison between a "shared map" approach and a "per-destination/per-head map" approach, keeping the total budget constant.

Evaluate LayerTrace on the full 13-task RULER benchmark and compare its performance against the existing baselines presented in Table 15, ensuring that the KV cache budget is matched.

Reviewer_pDbwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LayerTrace, which addresses the problem of sparse attention selection when historical keys and values are repeatedly read during long-context decoding. It leverages the latent for historical tokens in the source layer, and trains destination-layer and query-head scorers to determine the read maps for future layers. Evaluation on LongBench v2 shows LayerTrace accelerates the decoding while maintaining quality.

Critical strengths

This paper targets an important problem in long-context decoding: reducing the historical keys and values read during decoding by sparse attention. LayerTrace learns a token representation using earlier transformer layers to predict layer attention needs, which provides a motivated approach to achieve efficient sparse attention during long-context decoding.

Critical weaknesses
  1. The abstract fails to provide a clear framing of the central contribution of this paper, and the writing for the main content is hard to follow. For instance, the last three sentences, “A 56-question DeepSeek development diagnostic scores 59.08 versus 49.40, but all-selected numerical discrepancies remain unresolved. Higher thresholds can collapse quality, and single-request execution can remain slower. These exploratory observations do not yet establish quality-preserving speedups under matched batched answer generation.” does not convey a coherent point for the findings of the paper, and instead reads like content in an experiment report. In section 6, the second paragraph: “Qwen quality/timing share checkpoints, calibration and core kernels. Its quality runs pass 24/24 fast/reference and 4/4 native checks; separate full-model KL checks at B8/B16 are unavailable. DeepSeek fails 2/4 native and 7/24 fast/reference checks. Scores use actual fast-path generation; these observations do not confirm quality-preserving batched answer-generation speedups.” only lists experiment details without proper explanation and enough background information. This kind of writing makes the paper hard to follow and understand.

  2. The benefit of the proposed design using learned historical latent and independent scorers over simpler baselines is not clearly isolated. The differences between LayerTrace and the other baselines in execution details make the comparison on the quality-latency trade-off inappropriate.

Questions for authors
  1. In Section 6, the task quality and fixed-input decoder timing experiments adopt different implementation and batching conditions. Can the authors use a single experiment setting to justify the performance of LayerTrace under both task quality and decoder time metrics to support the paper’s central practical benefit?

  2. During the training of the token index and the scorers, equation (4) tries to reproduce dense attention distributions. The deployment instead uses hard thresholds to choose the masks and recomputes attention using the destination layer’s actual query. Can the authors explain why minimizing the surrogate loss in equation (4) could support a reliable selection of tokens during deployment?

  3. As indicated in the second critical weakness, can the authors isolate the contributions of the learned historical latent and independent scorers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.