- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses two obstacles to deploying block attention for long-context LLMs: the lack of principled automatic segmentation and the inefficiency of existing block fine-tuning. The authors construct SemanticSeg, a multi-domain segmentation dataset labeled with Gemini-2.5-Pro, and train a lightweight segmenter (Qwen3-4B or 0.6B backbone with a classification head) that predicts semantic cut points and supports controllable granularity. They then propose Block Distillation, a training framework that transfers full-attention capability to block-attention execution while preserving full-attention performance. Block Distillation introduces three components: block sink tokens to mitigate abnormal attention at block heads, block dropout to exploit training signals from all blocks via KL divergence, and token-level loss weighting to emphasize tokens most sensitive to block attention. Experiments on LongBench and LoCoMo across Qwen3-4B, Llama-3.1-8B, Qwen3-8B, and Qwen3-14B show that Block Distillation brings block-attention performance close to full attention, preserves or improves full-attention capability, and reduces TTFT substantially at long contexts. Training is about 26% faster than block fine-tuning.
Critical strengths
-
The paper addresses both automatic semantic segmentation and efficient training: it introduces the SemanticSeg dataset and a lightweight segmenter, and proposes Block Distillation, which brings block-attention performance close to full attention while preserving or even improving full-attention capability. This is consequential because prior block-attention work either relies on heuristic segmentation or on expensive block fine-tuning that generalizes poorly, making real-world adoption difficult.
-
Training is about 26% faster than block fine-tuning, and inference TTFT at 64k context drops from 6,971.3 ms to 3,821.6 ms; on LongBench and LoCoMo, block-attention performance approaches the full-attention upper bound. This shows that the method delivers measurable gains in realistic long-context settings, not just theoretical speedups, which directly supports acceptance.
Critical weaknesses
The experiments cover only LongBench and LoCoMo, with models up to 14B, and do not test MoE, agentic scenarios, thinking mode, or RL compatibility. The paper also discusses CacheBlend and others in related work but does not compare against them experimentally. This is consequential because it is unclear whether Block Distillation truly outperforms existing KV-reuse approaches, and the generality claims remain weakly supported. The ablation shows that removing token weights improves Multi-doc QA performance (42.87 vs. 37.55), suggesting the weighting scheme may hurt some tasks even if it helps others. This trade-off is not analyzed.
SemanticSeg is automatically labeled by a proprietary model, and the paper does not analyze how label noise or model bias affects the segmenter. Moreover, SAVB/RCEP calibration samples are not enforced to be fully disjoint from evaluation. This is consequential because it weakens reproducibility, fairness, and the evidence for out-of-distribution generalization, especially given the claim of cross-domain applicability.
Questions for authors
-
Can you evaluate Block Distillation on additional long-context benchmarks such as RULER, InfiniteBench, or HELMET, and on larger models (e.g., 32B+) to test generality?
-
How does Block Distillation compare experimentally against CacheBlend, KVLink, SpecCache, and C²KV under the same backbones and evaluation protocol?
-
Can you provide a fully controlled comparison with block fine-tuning on the same base model, same training data, and same segmentation, to isolate the effect of the distillation framework?
-
How sensitive is the segmenter to noise or bias in the Gemini-2.5-Pro labels? Can you report inter-annotator agreement or a human evaluation of a subset?
-
Why does removing token weights improve Multi-doc QA but degrade other tasks? Would task-adaptive or dynamic weighting be beneficial?
-
What are the throughput, memory usage, and latency under different batch sizes and generation lengths? Does the TTFT advantage translate into end-to-end speedups for realistic serving?
4 comments
Reply 2
The ablation shows that removing token weights improves Multi-doc QA performance (42.87 vs. 37.55), suggesting the weighting scheme may hurt some tasks even if it helps others. This trade-off is not analyzed.
Why does removing token weights improve Multi-doc QA but degrade other tasks? Would task-adaptive or dynamic weighting be beneficial?
We believe the effect is highly related to the coefficients and in Eq.2. Adjusting these coefficients has different effects on different tasks. However, the effect does not affect the overall gains and conclusions as long as the coefficients remain in a reasonable range. Therefore, we leave the exploration of task-adaptive or dynamic weighting to future work.
Dear Reviewer JbVk,
The CacheBlend and KVLink results for models other than Qwen3-8B have been updated in Table 3.
We would appreciate it if you could have a look at them and give us further feedback.
Thank you for your responses. My main concerns have been addressed, and I maintain my positive rating.
Reply 1
Dear Reviewer JbVK,
Thank you for your comments. We respond as follows.
Thank you for your suggestion. Due to the limited rebuttal period, we conducted experiments on RULER for Qwen3-8B, the results are as follows.
The performance gap between full attention (original - full) and block attention (block-dist - block) is relatively narrow.
Thank you for this suggestion. We have run the experiments on CacheBlend and KVLink for Qwen3-8B (Due to the limited rebuttal period, only Qwen3-8B is tested. We aim to complete experiments for other models in the revision), add update the results to Table 3. Generally, their performance is inferior to our method.
Thank you for this point, we ran experiments for Qwen3-8B following this setting. The results are added to Table 3. Generally, block-FT cannot reach the performance of vanilla full attention.
For SemanticSeg, we manually checked 10 instances for each category to ensure the segmentation labeled by Gemini is aligned with human expectations. This could generally eliminate model biases. For the segmenter, we checked its segmentation results (the segmented dataset used for training the block attention model) in detail. Moreover, we split a test set from the SemanticSeg and computed the F1 score of the segmentation from the trained segmenter. We found that the F1 score could reach 0.77 for Qwen3-4B-Instruct-2507. This proves that the segmenter follows the distribution of the SemanticSeg.
For the segmenter cost, we show in Table 1 that it is possible to shrink the size of the segmenter without affecting the performance too much (Table 1). Therefore, it is possible to further reduce the size of the segmenter so as to reduce the memory overhead.
In addition, the segmentation and block attention do not necessarily follow a one-to-one online manner. In many application scenarios like RAG, coding, etc, the segmentation can be safely decoupled from the later block attention process and be conducted in an offline manner. In this way, the segmenter is not directly involved in the block attention inference process, so the segmentation latency does not directly affect the block attention inference process. Therefore, the end-to-end latency cannot be verified by a simple "cut-then-block" sequential procedure; it requires more comprehensive metrics.
Here, we report the segmenter cost at single-pass. The recursion cost is approximately a multiple of this latency. In addition, we found that most segmentation can be fulfilled within two recursion levels.
Compared to the inference cost reported in Table 6, the latency of the 0.6B segmenter size is nearly 20%.