acceptodds
Under review as a conference paper at ICLR 2027

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

Abstract

Block attention, which processes the input as separate blocks that cannot attend to one another, offers significant potential to improve KV cache reuse in long-context scenarios such as Retrieval-Augmented Generation (RAG). However, its broader application is hindered by two key challenges: the difficulty of segmenting input text into meaningful, self-contained blocks, and the inefficiency of existing block fine-tuning methods that risk degrading performance. To address these, we first construct SemanticSeg, a large and diverse semantic segmentation dataset containing over instances across 16 categories—including books, code, web text, and conversations with text lengths ranging from to . Using this dataset, we train a lightweight segmenter to automatically partition text into human-instinct-aligned blocks with controllable granularity. Second, we propose block distillation, a training framework that is more efficient than block fine-tuning, which uses a frozen full-attention teacher model to guide the block-attention student. This framework integrates three novel components: block sink tokens to mitigate information loss at block boundaries, block dropout to leverage training signals from all blocks, and token-level loss weighting to focus learning on block-attention-sensitive tokens. Experiments across multiple models and benchmarks demonstrate that our segmenter outperforms heuristic and statistical baselines, and block distillation achieves near-full-attention performance under block attention, establishing a practical and scalable pathway for deploying block attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Block attention, Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_JbVkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses two obstacles to deploying block attention for long-context LLMs: the lack of principled automatic segmentation and the inefficiency of existing block fine-tuning. The authors construct SemanticSeg, a multi-domain segmentation dataset labeled with Gemini-2.5-Pro, and train a lightweight segmenter (Qwen3-4B or 0.6B backbone with a classification head) that predicts semantic cut points and supports controllable granularity. They then propose Block Distillation, a training framework that transfers full-attention capability to block-attention execution while preserving full-attention performance. Block Distillation introduces three components: block sink tokens to mitigate abnormal attention at block heads, block dropout to exploit training signals from all blocks via KL divergence, and token-level loss weighting to emphasize tokens most sensitive to block attention. Experiments on LongBench and LoCoMo across Qwen3-4B, Llama-3.1-8B, Qwen3-8B, and Qwen3-14B show that Block Distillation brings block-attention performance close to full attention, preserves or improves full-attention capability, and reduces TTFT substantially at long contexts. Training is about 26% faster than block fine-tuning.

Critical strengths
  1. The paper addresses both automatic semantic segmentation and efficient training: it introduces the SemanticSeg dataset and a lightweight segmenter, and proposes Block Distillation, which brings block-attention performance close to full attention while preserving or even improving full-attention capability. This is consequential because prior block-attention work either relies on heuristic segmentation or on expensive block fine-tuning that generalizes poorly, making real-world adoption difficult.

  2. Training is about 26% faster than block fine-tuning, and inference TTFT at 64k context drops from 6,971.3 ms to 3,821.6 ms; on LongBench and LoCoMo, block-attention performance approaches the full-attention upper bound. This shows that the method delivers measurable gains in realistic long-context settings, not just theoretical speedups, which directly supports acceptance.

Critical weaknesses

The experiments cover only LongBench and LoCoMo, with models up to 14B, and do not test MoE, agentic scenarios, thinking mode, or RL compatibility. The paper also discusses CacheBlend and others in related work but does not compare against them experimentally. This is consequential because it is unclear whether Block Distillation truly outperforms existing KV-reuse approaches, and the generality claims remain weakly supported. The ablation shows that removing token weights improves Multi-doc QA performance (42.87 vs. 37.55), suggesting the weighting scheme may hurt some tasks even if it helps others. This trade-off is not analyzed.

SemanticSeg is automatically labeled by a proprietary model, and the paper does not analyze how label noise or model bias affects the segmenter. Moreover, SAVB/RCEP calibration samples are not enforced to be fully disjoint from evaluation. This is consequential because it weakens reproducibility, fairness, and the evidence for out-of-distribution generalization, especially given the claim of cross-domain applicability.

Questions for authors
  1. Can you evaluate Block Distillation on additional long-context benchmarks such as RULER, InfiniteBench, or HELMET, and on larger models (e.g., 32B+) to test generality?

  2. How does Block Distillation compare experimentally against CacheBlend, KVLink, SpecCache, and C²KV under the same backbones and evaluation protocol?

  3. Can you provide a fully controlled comparison with block fine-tuning on the same base model, same training data, and same segmentation, to isolate the effect of the distillation framework?

  4. How sensitive is the segmenter to noise or bias in the Gemini-2.5-Pro labels? Can you report inter-annotator agreement or a human evaluation of a subset?

  5. Why does removing token weights improve Multi-doc QA but degrade other tasks? Would task-adaptive or dynamic weighting be beneficial?

  6. What are the throughput, memory usage, and latency under different batch sizes and generation lengths? Does the TTFT advantage translate into end-to-end speedups for realistic serving?

4 comments

Dear Reviewer JbVK,

Thank you for your comments. We respond as follows.

Moreover, SAVB/RCEP calibration samples are not enforced to be fully disjoint from evaluation. This is consequential because it weakens reproducibility, fairness, and the evidence for out-of-distribution generalization, especially given the claim of cross-domain applicability.

Can you evaluate Block Distillation on additional long-context benchmarks such as RULER, InfiniteBench, or HELMET, and on larger models (e.g., 32B+) to test generality?

Thank you for your suggestion. Due to the limited rebuttal period, we conducted experiments on RULER for Qwen3-8B, the results are as follows.

Method8k16k32k
Original - Full96.091.891.2
Block-Dist - Full95.590.990.0
Block-Dist - Block95.091.489.9

The performance gap between full attention (original - full) and block attention (block-dist - block) is relatively narrow.

The paper also discusses CacheBlend and others in related work but does not compare against them experimentally. This is consequential because it is unclear whether Block Distillation truly outperforms existing KV-reuse approaches, and the generality claims remain weakly supported.

How does Block Distillation compare experimentally against CacheBlend, KVLink, SpecCache, and C²KV under the same backbones and evaluation protocol?

Thank you for this suggestion. We have run the experiments on CacheBlend and KVLink for Qwen3-8B (Due to the limited rebuttal period, only Qwen3-8B is tested. We aim to complete experiments for other models in the revision), add update the results to Table 3. Generally, their performance is inferior to our method.

Can you provide a fully controlled comparison with block fine-tuning on the same base model, same training data, and same segmentation, to isolate the effect of the distillation framework?

Thank you for this point, we ran experiments for Qwen3-8B following this setting. The results are added to Table 3. Generally, block-FT cannot reach the performance of vanilla full attention.

SemanticSeg is automatically labeled by a proprietary model, and the paper does not analyze how label noise or model bias affects the segmenter.

How sensitive is the segmenter to noise or bias in the Gemini-2.5-Pro labels? Can you report inter-annotator agreement or a human evaluation of a subset?

For SemanticSeg, we manually checked 10 instances for each category to ensure the segmentation labeled by Gemini is aligned with human expectations. This could generally eliminate model biases. For the segmenter, we checked its segmentation results (the segmented dataset used for training the block attention model) in detail. Moreover, we split a test set from the SemanticSeg and computed the F1 score of the segmentation from the trained segmenter. We found that the F1 score could reach 0.77 for Qwen3-4B-Instruct-2507. This proves that the segmenter follows the distribution of the SemanticSeg.

What are the throughput, memory usage, and latency under different batch sizes and generation lengths? Does the TTFT advantage translate into end-to-end speedups for realistic serving?

For the segmenter cost, we show in Table 1 that it is possible to shrink the size of the segmenter without affecting the performance too much (Table 1). Therefore, it is possible to further reduce the size of the segmenter so as to reduce the memory overhead.

In addition, the segmentation and block attention do not necessarily follow a one-to-one online manner. In many application scenarios like RAG, coding, etc, the segmentation can be safely decoupled from the later block attention process and be conducted in an offline manner. In this way, the segmenter is not directly involved in the block attention inference process, so the segmentation latency does not directly affect the block attention inference process. Therefore, the end-to-end latency cannot be verified by a simple "cut-then-block" sequential procedure; it requires more comprehensive metrics.

Here, we report the segmenter cost at single-pass. The recursion cost is approximately a multiple of this latency. In addition, we found that most segmentation can be fulfilled within two recursion levels.

LengthTime (ms)
Qwen3-4B-Instruct-2507 - 32k32k1171.8
Qwen3-0.6B - 32k32k217.4

Compared to the inference cost reported in Table 6, the latency of the 0.6B segmenter size is nearly 20%.

The ablation shows that removing token weights improves Multi-doc QA performance (42.87 vs. 37.55), suggesting the weighting scheme may hurt some tasks even if it helps others. This trade-off is not analyzed.

Why does removing token weights improve Multi-doc QA but degrade other tasks? Would task-adaptive or dynamic weighting be beneficial?

We believe the effect is highly related to the coefficients α\alpha and β\beta in Eq.2. Adjusting these coefficients has different effects on different tasks. However, the effect does not affect the overall gains and conclusions as long as the coefficients remain in a reasonable range. Therefore, we leave the exploration of task-adaptive or dynamic weighting to future work.

Dear Reviewer JbVk,

The CacheBlend and KVLink results for models other than Qwen3-8B have been updated in Table 3.

We would appreciate it if you could have a look at them and give us further feedback.

Thank you for your responses. My main concerns have been addressed, and I maintain my positive rating.

Reviewer_tsrpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to extend block attention, which enables independent context encoding and KV-cache reuse, to inputs without predefined document boundaries. It introduces SemanticSeg, a Gemini-labeled semantic segmentation dataset spanning 16 categories, and trains a segmenter to select boundaries with controllable granularity. It also proposes block distillation from a frozen full-attention teacher, combining block sink tokens, block dropout for denser KL supervision, and token-level loss weighting. Experiments on LongBench and LoCoMo across four Qwen and Llama models report block-attention performance close to the original full-attention models. The paper additionally evaluates segmentation alternatives, ablates the training components, and reports training-step and time-to-first-token measurements.

Critical strengths
  1. The reported recovery of block-attention performance is consistent across four models and two benchmarks (Tables 3–4). For example, Qwen3-4B reaches 40.86 on LongBench under block attention versus 39.63 for the original full-attention model, while Llama-3.1-8B reaches 36.32 versus 36.75. This supports the practical usefulness of the adaptation recipe beyond a single model or task.

  2. The component ablations provide meaningful evidence for the training design. On Qwen3-4B, removing block dropout lowers the LongBench average from 40.86 to 34.49, and removing KL supervision lowers it to 32.13 (Table 5). These results support the importance of denser distillation supervision. The segmentation comparison also controls the number of blocks and evaluates models not trained with the learned segmenter, addressing plausible confounds in that part of the evaluation.

Critical weaknesses
  1. Several reported averages are inconsistent with the displayed scores. The Qwen3-8B full-attention row has identical component scores in Tables 1 and 3, but its average is reported as 37.85 and 36.05, respectively; the arithmetic mean is 36.05. The Tulu3 segmenter-4B row averages to 31.83 rather than 32.82, with the same discrepancy repeated in Table 2. The Tulu3 Random row in Table 1 also averages to 28.22 rather than 30.38. Recomputing the displayed rows preserves the segmenter's leading average but changes the reported margins. It is unclear whether the errors are confined to the Average columns or also involve the component scores.

  2. The training comparison does not match initialization and training history between the Tulu3 block-fine-tuning baselines and the distilled models. Continuing Tulu3-Block-FT on the segmented data does not fully isolate the effect of the training method. Table 6 reports per-step training time without establishing total training cost at comparable quality. The inference measurements also leave the block configuration, cache-hit conditions, and inclusion or amortization of segmentation, cache construction, and cache loading unspecified.

Questions for authors
  1. Reconciliation of the numerical results: Please explain why the identical Qwen3-8B full-attention component scores are assigned averages of 37.85 in Table 1 and 36.05 in Table 3, and why the Tulu3 segmenter-4B scores average to 31.83 rather than the 32.82 reported in Tables 1–2. Please state the aggregation rule, reconcile the affected entries across these tables, and clarify whether the discrepancies are confined to the Average columns or also involve the component scores. Do the corrected results change the claimed segmentation gains or the generalization comparison?

  2. Clarification of the training comparison: Please specify the initialization, training data, and optimization budget used for Tulu3-Block-FT-S relative to the distilled models. Does any existing comparison hold these factors fixed? If not, what evidence supports attributing the remaining gap specifically to the fine-tuning method rather than to the differing training histories?

  3. Clarification of the efficiency measurements: For Table 6, what are the block count/length distribution and cache-hit conditions, and which stages are included in TTFT? In particular, are segmentation, block prefilling, and cache loading counted or amortized? For the training comparison, please state the batch/token budget and number of optimization steps for each method.

4 comments

Dear Reviewer tsrp,

Thank you very much for your comments. We respond as follows.

Several reported averages are inconsistent with the displayed scores. The Qwen3-8B full-attention row has identical component scores in Tables 1 and 3, but its average is reported as 37.85 and 36.05, respectively; the arithmetic mean is 36.05. The Tulu3 segmenter-4B row averages to 31.83 rather than 32.82, with the same discrepancy repeated in Table 2. The Tulu3 Random row in Table 1 also averages to 28.22 rather than 30.38. Recomputing the displayed rows preserves the segmenter's leading average but changes the reported margins. It is unclear whether the errors are confined to the Average columns or also involve the component scores.

Reconciliation of the numerical results: Please explain why the identical Qwen3-8B full-attention component scores are assigned averages of 37.85 in Table 1 and 36.05 in Table 3, and why the Tulu3 segmenter-4B scores average to 31.83 rather than the 32.82 reported in Tables 1–2. Please state the aggregation rule, reconcile the affected entries across these tables, and clarify whether the discrepancies are confined to the Average columns or also involve the component scores. Do the corrected results change the claimed segmentation gains or the generalization comparison?

Thank you for pointing out this error. We apologize for our negligence. The inaccurate computation for average metrics results from inappropriate decimal handling during the aggregation (The aggregation was conducted using full decimals, but the reported component scores are shown in just two decimals). The inconsistent table results is because of the sampling randomness in the model generation (we initiate separate runs for different tables). We have corrected all these errors, and updated our submission. We also double-checked all component scores and found no mistakes. The corrected results do not affect the conclusion and the observed gains.

The training comparison does not match initialization and training history between the Tulu3 block-fine-tuning baselines and the distilled models. Continuing Tulu3-Block-FT on the segmented data does not fully isolate the effect of the training method.

Clarification of the training comparison: Please specify the initialization, training data, and optimization budget used for Tulu3-Block-FT-S relative to the distilled models. Does any existing comparison hold these factors fixed? If not, what evidence supports attributing the remaining gap specifically to the fine-tuning method rather than to the differing training histories?

For the training comparison, the reason for using the continuing training scheme is that we cannot reproduce the randomly sampled RAG training data used in the training of the original block-FT. But we believe your point is of great value, therefore we run further experiment of training the Tulu3-SFT using both the block-FT training data and our segmented data. The result has been added to Table 2. (Row Tulu3-Block-FT-SS - Block).

In addition, we also reported the block-FT result for Qwen3-8B (Due to the limited rebuttal period, only Qwen3-8B is tested. We aim to complete experiments for other models in the revision) using only our segmented data (Table 3). The results further demonstrate the superiority of our distillation methods.

Table 6 reports per-step training time without establishing total training cost at comparable quality. The inference measurements also leave the block configuration, cache-hit conditions, and inclusion or amortization of segmentation, cache construction, and cache loading unspecified.

Clarification of the efficiency measurements: For Table 6, what are the block count/length distribution and cache-hit conditions, and which stages are included in TTFT? In particular, are segmentation, block prefilling, and cache loading counted or amortized? For the training comparison, please state the batch/token budget and number of optimization steps for each method.

Thank you for this point. For the training comparison in Table 6, we run 50 optimization steps for both methods and report the average. The batch number is set to 8, and the training data length is 16k. Therefore, the token budget is 8×16k×50=6.4m8 \times 16k \times 50 = 6.4m.

For the inference measurement, we create an input by concatenating several 2k-length documents into a predefined-length input (length in Table 6). The KVs are prefilled before the measurement for both vanilla full attention and block attention. Then we measure the TTFT on a new random input with the same set of documents. The segmentation and prefilling stage are amortized.

Dear Reviewer tsrp,

The block-FT results for models other than Qwen3-8B are updated in Table 3.

We would appreciate it if you could take a moment to review them.

Thank you for your response. The clarifications have addressed my main concerns, and I will raise my score accordingly.

Thank you very much for the time and effort devoted to evaluating our paper. We greatly appreciate the detailed and constructive comments, which have helped us improve this work.

Reviewer_NnofOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to make block attention practical for long-context and RAG workloads by solving two obstacles: (1) the difficulty of segmenting text into coherent, self-contained blocks, and (2) the cost and limited generalization of existing block fine-tuning. It introduces SemanticSeg, a large multi-category segmentation dataset, and trains a lightweight neural segmenter (based on Qwen3-4B) that produces adaptive boundaries with controllable granularity via thresholds and recursion. It further proposes block distillation, in which a frozen full-attention teacher guides a block-attention student, augmented by three techniques: block sink tokens to stabilize block-head attention, block dropout to extract training signals from all blocks, and token-level loss weighting that emphasizes tokens sensitive to the attention pattern change. The authors report that the segmenter outperforms heuristic baselines and that block distillation approaches full-attention quality while being more efficient than prior block fine-tuning.

Critical strengths

The problem is practically important. Enabling prefix-agnostic KV cache reuse for overlapping documents in RAG can yield substantial compute savings, yet naïve block attention degrades performance sharply. Addressing both the segmentation barrier and the training barrier is a well-motivated goal. SemanticSeg and the learned segmenter constitute a clear advance over newline or rule-based splitting. The dataset’s diversity (16 categories, 2k–32k lengths) and the controllable granularity mechanism (threshold + recursive application) are useful engineering contributions. The distillation framework is thoughtfully designed. Using a frozen teacher avoids the dual full/block update schedule of prior work; block sinks target the known “lost-in-block-head” pathology; block dropout increases the density of usable supervision; and token weighting focuses capacity on positions where the attention pattern change matters most. These components are complementary and well-motivated. The overall pipeline (automatic segmentation → block distillation → near-full-attention block inference) offers a concrete, scalable path toward deploying block attention beyond narrowly curated RAG settings.

Critical weaknesses

Dataset quality and potential bias are under-examined. SemanticSeg is constructed by prompting Gemini-2.5-Pro on heuristically inserted candidate cuts. The paper does not report inter-annotator agreement, human validation rates, or analysis of systematic biases the teacher model may introduce (e.g., preference for paragraph- or section-level cuts). This is important because the segmenter’s downstream utility rests on the fidelity of these labels. The residual performance gap and failure modes are not fully characterized. “Near-full-attention” is a strong claim; the main text should quantify the remaining gap on tasks that require cross-block reasoning (multi-hop QA, long-range coreference, code with distant dependencies). Cases in which semantically coherent blocks still omit critical cross-block context should be analyzed. Efficiency accounting is incomplete. While block distillation is claimed to be cheaper than block fine-tuning, the overhead of running the segmenter (especially with recursion) at inference time is not reported, nor is the end-to-end latency or memory benefit under realistic RAG cache-hit rates. Controllable granularity is useful, but the accuracy–parallelism trade-off curve is only lightly explored. Several design choices remain somewhat ad-hoc. The decision to duplicate the sink token four times, the specific dropout rate (~0.6), and the exact form of the token weights (Eq. 2) lack sensitivity analysis in the main text. Ablations that isolate each of the three components are essential and should be presented prominently. Generalization beyond the training distribution of SemanticSeg is only partially demonstrated. Performance on domains or languages absent from the 16 categories, or on texts whose optimal block boundaries are highly task-dependent rather than purely semantic, remains an open question.

Questions for authors

What fraction of SemanticSeg labels were manually verified, and how does segmenter performance correlate with label confidence or category? On the hardest cross-block reasoning tasks, how large is the remaining gap between the distilled block-attention model and the full-attention teacher? Are there systematic error patterns? What is the wall-clock and memory overhead of the segmenter (single-pass vs. recursive) relative to the savings obtained from KV reuse under typical RAG hit rates? Can you provide full ablations that remove sink tokens, block dropout, and token weighting one at a time, both for final block-attention accuracy and for training stability?

2 comments

Dear Reviewer Nnof,

Thank you for your comments. We respond as follows.

Dataset quality and potential bias are under-examined. SemanticSeg is constructed by prompting Gemini-2.5-Pro on heuristically inserted candidate cuts. The paper does not report inter-annotator agreement, human validation rates, or analysis of systematic biases the teacher model may introduce (e.g., preference for paragraph- or section-level cuts). This is important because the segmenter’s downstream utility rests on the fidelity of these labels.

What fraction of SemanticSeg labels were manually verified, and how does segmenter performance correlate with label confidence or category?

For SemanticSeg, we manually checked 10 instances for each category to ensure the segmentation labeled by Gemini is aligned with human expectations. This could generally eliminate model biases. For the segmenter, we checked its segmentation results (the segmented dataset used for training the block attention model) in detail. Moreover, we split a test set from the SemanticSeg and computed the F1 score of the segmentation from the trained segmenter. We found that the F1 score could reach 0.77 for Qwen3-4B-Instruct-2507. This proves that the segmenter follows the distribution of the SemanticSeg. These details have been added to Appendix B.1 and C.1.

The residual performance gap and failure modes are not fully characterized. “Near-full-attention” is a strong claim; the main text should quantify the remaining gap on tasks that require cross-block reasoning (multi-hop QA, long-range coreference, code with distant dependencies). Cases in which semantically coherent blocks still omit critical cross-block context should be analyzed.

On the hardest cross-block reasoning tasks, how large is the remaining gap between the distilled block-attention model and the full-attention teacher? Are there systematic error patterns?

Many reported tasks require cross-block reasoning, including the Multi-doc QA, Code in Table 3, and Multi-hop, Temporal, Open domain in Table 4, etc. Overall, there is still a narrow gap between full attention and block attention, which indicates the training may require further scaling (Discussed in the limitations). We consider weakening our claim based on your point.

Efficiency accounting is incomplete. While block distillation is claimed to be cheaper than block fine-tuning, the overhead of running the segmenter (especially with recursion) at inference time is not reported, nor is the end-to-end latency or memory benefit under realistic RAG cache-hit rates.

What is the wall-clock and memory overhead of the segmenter (single-pass vs. recursive) relative to the savings obtained from KV reuse under typical RAG hit rates?

For the segmenter cost, we show in Table 1 that it is possible to shrink the size of the segmenter without affecting the performance too much (Table 1). Therefore, it is possible to further reduce the size of the segmenter so as to reduce the memory overhead.

In addition, the segmentation and block attention do not necessarily follow a one-to-one online manner. In many application scenarios like RAG, coding, etc., the segmentation can be safely decoupled from the later block attention process and be conducted in an offline manner. In this way, the segmenter is not directly involved in the block attention inference process, so the segmentation latency does not directly affect the block attention inference process. Therefore, the end-to-end latency cannot be verified by a simple "cut-then-block" sequential procedure; it requires more comprehensive metrics.

Here, we report the segmenter cost at single-pass. The recursion cost is approximately a multiple of this latency. In addition, we found that most segmentation can be fulfilled within two recursion levels.

LengthTime (ms)
Qwen3-4B-Instruct-2507 - 32k32k1171.8
Qwen3-0.6B - 32k32k217.4

Compared to the inference cost reported in Table 6, the latency of the 0.6B segmenter size is nearly 20%.

Controllable granularity is useful, but the accuracy–parallelism trade-off curve is only lightly explored. Several design choices remain somewhat ad-hoc. The decision to duplicate the sink token four times, the specific dropout rate (~0.6), and the exact form of the token weights (Eq. 2) lack sensitivity analysis in the main text.

Thank you for this point. For the accuracy-parallelism trade-off, we run different numbers of recursion levels for the segmenter-4B on the Multi-doc QA task in Longbench. The results are as follows:

Recursion Level12
Multi-doc QA41.7739.95
Average number of blocks12.2417.37

This provides a brief view of the trade-off.

The sink token duplication rate is determined by following the setting of the original attention sink paper. We believe your proposed points are of great value and tend to dive into them in more detail in the future.

Ablations that isolate each of the three components are essential and should be presented prominently. Generalization beyond the training distribution of SemanticSeg is only partially demonstrated. Performance on domains or languages absent from the 16 categories, or on texts whose optimal block boundaries are highly task-dependent rather than purely semantic, remains an open question.

Can you provide full ablations that remove sink tokens, block dropout, and token weighting one at a time, both for final block-attention accuracy and for training stability?

The ablation experiments are provided in Section 5.3 and Table 5, including the three components in block distillation.

To test the generality, we conduct experiments on RULER for Qwen3-8B. We believe it fulfills your proposed requirements (block boundaries are highly task-dependent rather than purely semantic, and it differs from the distribution of SemanticSeg). The results are as follows.

Method8k16k32k
Original - Full96.091.891.2
Block-Dist - Full95.590.990.0
Block-Dist - Block95.091.489.9

The performance gap between full attention (original - full) and block attention (block-dist - block) is relatively narrow.

Reviewer_oEX3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • the paper builds on top of block attention models: these are models that enable independently encoded, reusable KV caches
  • the paper introduces a new training recipe to train block attention models
  • the authors first train a segmenter that decides how to chunk the large context that the subsequent block attention model consumes
  • using the blocks as defined by the segmenter, the authors then distill (instead of finetuning) on the logits provided by the teacher
  • This enables independently encoded, reusable KV caches while largely preserving original performance
Critical strengths
  • the method proposed in the paper not only recovers original full attention performance when using blocked prefills
  • at the same time, it preserves performance when the same finetuned model is used as a full attention model
  • block distillation seem to be better than just vanilla block finetuning -- where performance drops if ones removes either block dropout or KL loss
Critical weaknesses
  • i feel a block finetuned qwen baseline is missing -- right now the paper evaluates block finetuning on Tulu, but does block distillation on Qwen checkpoints -- there's some apple to oranges comparison happening
  • i'm not convinced by the importance of segmenter (probably one doesn't need it) -- the improvement in qwen over random candidate segmentation is only 18.32 versus 18.21 in table 1
Questions for authors
  • does block distillation outperform block fine-tuning when both start from the same Qwen or Tulu checkpoint ? if not, then the gains are probably due to the better pretrained qwen model
  • how do the block distilled models perform with simple segmentation (rule based say) compared with the learned segmenter? if they perform similarly, there's no point of the segmenter, and the gains are only from the distillation/training recipe
  • a line of related work is missing that compress the context end to end, and can in principle be used to address the KV shift problem: https://arxiv.org/abs/2406.02657, https://arxiv.org/abs/2511.05313, https://arxiv.org/abs/2606.09659 etc.
4 comments

Dear Reviewer oEX3,

Thank you for your comments. We respond as follows.

i feel a block finetuned qwen baseline is missing -- right now the paper evaluates block finetuning on Tulu, but does block distillation on Qwen checkpoints -- there's some apple to oranges comparison happening

does block distillation outperform block fine-tuning when both start from the same Qwen or Tulu checkpoint ? if not, then the gains are probably due to the better pretrained qwen model

Thank you for this point. We have run an experiment for Qwen3-8B, and reported the results in Table 3 (Due to the limited rebuttal period, only Qwen3-8B is tested. We aim to complete experiments for other models in the revision). The results further demonstrate the validation of block distillation.

i'm not convinced by the importance of segmenter (probably one doesn't need it) -- the improvement in qwen over random candidate segmentation is only 18.32 versus 18.21 in table 1

how do the block distilled models perform with simple segmentation (rule based say) compared with the learned segmenter? if they perform similarly, there's no point of the segmenter, and the gains are only from the distillation/training recipe

Thank you for this point. We would like to emphasize that the segmenter provides a general approach in segmentation, which cannot be achieved by any rule. This is really important, as we target general application scenarios. The relatively small difference among variant segmentation baselines in Qwen is because it is not trained for block attention. On the contrary, the performance variance of Tulu3-Block-FT is relatively noticeable.

To further clarify, we conducted experiments on different segmentation approaches on the block-distilled models. The results are added to Table 1. The performance gap between different segmentation baselines is noticeable.

a line of related work is missing that compress the context end to end, and can in principle be used to address the KV shift problem: https://arxiv.org/abs/2406.02657, https://arxiv.org/abs/2511.05313, https://arxiv.org/abs/2606.09659 etc.

Thank you for the reminder. We will add a discussion of these to our related work.

Dear Reviewer oEX3,

We have updated the block-FT experiments for models other than Qwen3-8B in Table 3.

We would appreciate it if you could take some time to have a look.

sorry for the delay, and thanks for the response! my worries are mostly addressed, so i'm raising the score. all the best! :)

Thank you very much for your valuable advice, which has greatly improved this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.