acceptodds
Under review as a conference paper at ICLR 2027

MaskAhead: Unifying KV Cache Selection and Eviction in Block Diffusion Language Models

Abstract

Block diffusion language models decode one block at a time against an exact key-value cache of the prefix, which grows with the prompt and with the model's own long generations. A small cache requires two decisions: which prefix entries the current block should read, and which entries later blocks may need. Selection is temporary. Eviction permanently removes entries from storage. Sparse-read methods cut repeated cache reads but keep the full prefix resident. Eviction reduces storage, but the decision is irreversible: entries removed now cannot serve later blocks. Block diffusion exposes masked positions in upcoming blocks before generation, so a no-write probe can obtain model-derived queries for retention without producing draft tokens. MaskAhead assigns the model's mask queries to the decisions they can inform: current-block queries rank temporary reads, while a no-write probe of upcoming blocks ranks the entries retained until the next eviction. Both decisions use an output-sensitive score that accounts for values. The surviving cache can also be stored at four bits. On MATH-500, Q-MaskAhead reduces peak KV by 75%, with a controlled accuracy degradation. For long-prompt QA, question-conditioned prefill reduces peak KV by 86.7% on average across HotpotQA, NarrativeQA, and MuSiQue.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, KV cache, quantization, selection, inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bfK6OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaskAhead, a training-free KV-cache management method for block diffusion language models. Its central observation is that the model's own mask-token queries can inform two distinct decisions: temporary cache selection for the current block and irreversible cache eviction for future blocks. MaskAhead uses current-block queries for sparse reads and a no-write probe of upcoming masked blocks to estimate which prefix entries should be retained. Both decisions employ a CAOTE-inspired value-aware attention score. The authors further introduce Q-MaskAhead, which combines recurrent eviction with four-bit KV quantization, performing both ranking and attention directly from the quantized cache.

Experiments on Fast-dLLM-v2-7B, supplemented by preliminary cross-model results, cover mathematical reasoning, long-context question answering, retrieval, and serving efficiency. The method achieves substantial KV-memory reductions while retaining much, but not all, of dense-cache performance. The strongest gains occur under restrictive retention budgets, whereas competing methods become competitive or superior at wider budgets.

Critical strengths

I think one strength of the paper is that it makes a useful distinction between reducing cache reads and reducing memory storage. The proposed method uses information from both current and future blocks to make these decisions separately. I find the idea of looking ahead without storing extra KV states particularly interesting. It also doesn't require additional training or a separate prediction model, which makes the approach quite practical.

Another strength is that the paper provides fairly detailed experiments and ablation studies. The authors compare different query sources, scoring methods, memory budgets, and quantization settings. One encouraging result is that, with a cache capacity of 128, MaskAhead achieves 51% accuracy on MATH-500, compared with 39.4% for H2O. The authors also report statistical evidence supporting this improvement. I also appreciate that they discuss negative results, including limited benefits from value-aware scoring and weaker performance on some other models. Although I'm not an expert in this specific area, I think this makes the evaluation more balanced and convincing.

Critical weaknesses

My first concern is that the benefit of using future-mask queries has not been fully demonstrated.

I also have some concerns about the choice of the lookahead horizon. Once a cached entry is removed, it cannot be recovered later. So, looking ahead only until the next eviction checkpoint might miss information that becomes important further in the future.

I think the authors should explain more clearly why this horizon is appropriate and how it relates to the long-term importance of cached information.

Questions for authors

How sensitive is the method to the lookahead horizon? Since random-token probes perform almost as well, how can we tell whether the improvement really comes from future-mask queries?

Once a cache entry is removed, it cannot be recovered. So why is looking ahead only to the next eviction checkpoint enough? Could you also clarify how the current block is handled during eviction?

How does Q-MaskAhead compare with other methods, such as DapQ and H2O+CAOTE, when they all use the same four-bit quantization? Would the improvement still hold under the same memory and runtime budgets?

Could you explain how question-conditioned prefill is implemented? Does the reported memory usage include the extra memory and computation needed for probing? How would other retention methods perform with the same strategy?

How many eviction steps actually happen in your experiments? Could you report the output lengths and number of eviction checkpoints? This would help us understand whether the gains come from repeated lookahead decisions or mainly from initial cache compression. Could you also clarify how reproducible the preliminary DreamReasoner and LLaDA results are?

Reviewer_HnhKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MaskAhead, a KV cache management method for block diffusion LLMs that combines sparse read selection with permanent cache eviction. Current-block mask queries select the prefix KV entries to read, while a no-write probe of upcoming masked blocks estimates which entries to retain for future use. Both decisions use value-aware scores, and a quantized variant further reduces memory usage through four-bit KV storage.

Critical strengths
  1. Addresses an important problem. The paper addresses the growing KV cache memory footprint in block diffusion LLMs by reducing the number of stored KV entries in GPU.
  2. Uses future-position mask queries for retention. The method exploits a useful property of block diffusion LLMs: mask queries can be constructed at future positions before actual generation. These queries provide a signal for estimating which prefix KV entries may be needed by upcoming blocks.
  3. Includes a broad baseline comparison. The evaluation compares MaskAhead with multiple existing methods covering both sparse read selection and KV cache eviction.
Critical weaknesses
  1. Organization could be improved. The paper is sometimes difficult to follow because of its organization. For example, the explicit contribution list appears at the end of the Related Work section. Presenting this list in the Introduction would help readers identify the main contributions earlier.

  2. Insufficient justification for irreversible eviction. First, although mask queries at future positions can be computed before generation, the queries that actually arise in future blocks are conditioned on the intervening generated tokens. HERALD, for example, uses a draft of the current block to condition selection for the next block. MaskAhead probes future blocks without such draft conditioning. Although query-source ablations are provided, they do not directly quantify how closely the predicted retention scores match those obtained from actual future queries after the intervening blocks have been generated.

    Second, low importance over a few upcoming blocks does not establish that a prefix entry will remain unimportant throughout the rest of generation. An entry that is unnecessary in the near term may become relevant later, but permanent eviction prevents it from being retrieved again. Refreshing the retention decision at subsequent checkpoints cannot recover previously deleted entries. The paper therefore needs stronger evidence that its limited probe horizon is sufficient to guide irreversible eviction.

  3. Limited validation under long output generation. Evaluating permanent KV eviction requires sufficiently long outputs to test repeated eviction and the delayed reuse of earlier information. The HotpotQA probe ablation uses a 32-token output protocol, while the serving study generates 256 tokens per request. The output-length limits and actual generation-length distributions for the reasoning benchmarks are not clearly reported. Evaluation across longer outputs, together with the number of eviction events per request, would help establish whether the method preserves information needed well beyond the probe horizon.

Questions for authors
  1. Validation under long output generation. Could you evaluate MaskAhead on workloads with sufficiently long outputs to test repeated eviction and delayed reuse of earlier information?

  2. Agreement between predicted and actual future KV importance. How closely do predicted top-C retention sets match those computed from the corresponding blocks’ actual queries, using the same scoring rule and candidate keys? At a fixed retention budget, how does this agreement vary with the number of probed blocks and the evaluated block’s distance from the eviction checkpoint?

  3. Justification for irreversible eviction. What evidence supports permanently deleting entries based on a limited future-query horizon, given that they may become important again later in generation?

Reviewer_QkbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MaskAhead, a training-free KV cache management framework for block diffusion language models. Current-block mask queries select entries for temporary reads, while future-block mask queries guide irreversible cache eviction. Both decisions use a value-aware attention-output sensitivity score. Q-MaskAhead additionally uses four-bit KV storage for scoring and attention.

Experiments on reasoning, long-context question answering, and retrieval demonstrate competitive quality–memory trade-offs in some restrictive-budget settings, although the benefits vary across tasks, models, and budgets.

Critical strengths

The distinction between temporary read selection and irreversible eviction is useful, and assigning different query horizons to these decisions is well motivated.

Future-position mask queries avoid training an auxiliary predictor. Controlled experiments provide initial support, including a statistically supported improvement over DapQ-filled probes on MATH-500.

The paper examines scoring rules, probe horizons, eviction intervals, protected positions, and precision, while acknowledging settings where gains are inconclusive or competing methods perform better.

Separating prefill-peak memory from generation-end KV storage clarifies practical benefits. The paper also distinguishes ranking stability from preservation of answer quality.

Critical weaknesses

Limited independent novelty. Most components build on existing methods. The distinctive benefit of native future-mask queries over position-aligned pseudo queries is not sufficiently established.

Incomplete horizon-matching justification. Setting M = E overlooks irreversible information loss: entries needed beyond the probe horizon cannot be recovered at later checkpoints.

Insufficient validation of the scoring proxy. Query pooling, omitted renormalization, and independent token ranking may poorly approximate joint deletion error. Prefix-only normalization may also misrepresent actual attention sensitivity.

Unclear advantage over strong baselines. Gains over DapQ are small or inconsistent. Missing equally quantized baselines and matched-cost comparisons make the benefit of additional probing difficult to assess.

Implementation and reporting ambiguities. The prefill strategy is underspecified, measured and forecast results are not clearly separated, and inconsistent variant names and quantization numbers weaken reproducibility.

Questions for authors

How does MaskAhead protect information needed beyond the probe horizon, and does the optimal M consistently track E?

How well does Equation (3) predict actual attention-output error after joint eviction, including attention to the current block?

Does MaskAhead outperform DapQ and H2O+CAOTE under the same K4V4 format and memory budget when probe overhead is included?

Can you clarify the prefill configuration, distinguish measured from forecast results, and reconcile the inconsistent quantization recovery numbers?

Comments on the paper

1 comment

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.