acceptodds
Under review as a conference paper at ICLR 2027

Affix Cache for Diffusion Large Language Models

Abstract

Diffusion Large Language Models (DLLMs) enable non-autoregressive decoding, but efficient inference support remains immature: unlike autoregressive models, whose requests reuse a shared prefix key-value (KV) cache, DLLMs use bidirectional attention, so a shared context's KV states depend on the tokens still being decoded, leaving directly reused caches stale and full recomputation necessary. We present **ACache**, a cross-request cache reuse mechanism for shared spans, or *affixes*, at any position: prefix, infix, or suffix. ACache measures the influence of affix tokens on the masked generation region to identify a small request-specific subset as *Anchor Tokens*, and recomputes only their KV states while reusing the remaining affix cache. Built on state-of-the-art intra-request caching mechanisms, ACache recovers most of the accuracy lost to direct affix-cache reuse on average when recomputing around 20% of affix tokens, and at that budget preserves more accuracy than selection criteria adapted from prior cross-request cache-reuse systems. We co-design ACache with a modern inference engine, whose attention reads each request's recomputed Anchor KV states alongside one affix cache shared across concurrent requests. Against the same system with only intra-request caching, ACache cuts recompute latency by up to 56.7%, translating to as much as 1.71× end-to-end throughput, while reducing peak KV cache memory by up to 45.8%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion Language Models, Efficient Inference, KV Cache Reuse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AM2HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ACache enables cross-request KV cache reuse for shared text spans—including prefixes, infixes, and suffixes—in diffusion language models with bidirectional attention. It selects a small subset of Anchor Tokens for recomputation using mask-to-affix attention, while sharing the remaining KV states across requests. Integrated with dKV-Cache and Fast-dLLM, ACache reduces recomputation latency by up to 56.7%, achieves up to 1.71× end-to-end throughput, and reduces peak KV cache memory by up to 45.8% in the main few-shot experiments. Accuracy preservation varies across models and tasks.

Critical strengths
  1. The method extends cross-request cache reuse to different text positions in DLLMs, addressing the request-dependent nature of shared KV states under bidirectional attention.
  2. The system design has practical value: the read-slot map combines shared and private KV states without duplicating the entire affix cache for each request.
  3. The evaluation includes accuracy–efficiency trade-offs, Anchor stability analysis, and random-selection ablations, supporting the method’s effectiveness in the evaluated settings.
Critical weaknesses
  1. The algorithmic novelty needs further clarification. CacheBlend already explores selective recomputation, MaskKV uses mask-based attention scores for token importance, and concurrent work BiCache studies cross-request prefix reuse in DLLMs. Although ACache contributes support for different affix positions and a practical system design, the additional novelty of its core selection strategy remains insufficiently clear.
  2. The evaluation scope remains limited. Experiments cover only 7B/8B models, generally use 256-token generation, and reuse a fixed shared affix within each configuration. Effectiveness on larger models, longer generations, and serving workloads with multiple affixes or low cache hit rates remains unverified.
Questions for authors
  1. What is ACache’s key algorithmic contribution beyond CacheBlend’s selective recomputation, MaskKV’s mask-based attention scoring, and BiCache’s cross-request cache reuse?
  2. Why should mask-to-affix attention magnitude predict the benefit of recomputing a token? Could you validate the relationship between the scores and actual accuracy recovery, and justify normalizing only over affix tokens and aggregating equally across layers and heads?
  3. The current findings mainly concern 7B/8B models and 256-token generation. Does selecting Anchors only once remain reliable for larger models, longer contexts, and longer generations? Could you compare the quality and overhead of fixed versus periodically updated Anchors?
  4. With multiple affixes, varying cache hit rates, and cold starts, how many requests must reuse an affix to achieve a net benefit after accounting for cache precomputation? Could you characterize the break-even conditions in terms of shared-span length, concurrency, and cache hit rate?
Reviewer_LZbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses cross-request KV cache reuse in DLLMs by selectively recomputing the KV states of important tokens while reusing the shared KV cache for the remaining tokens. The authors also develop a dedicated inference engine and evaluate the method's accuracy and efficiency across multiple models and tasks.

Critical strengths

The paper introduces several system-level designs tailored to the proposed method to achieve practical speedups and reduce memory usage. I find these system designs interesting.

Critical weaknesses

The effectiveness of ACache is not sufficiently demonstrated. The comparisons with BiCache, the most directly relevant baseline, involve different inference engines and runtime configurations. A comparison under a unified system setup may be necessary to attribute the reported improvements to ACache itself. In addition, the paper reports accuracy (Table 5) and throughput (Table 6) separately, without providing Accuracy–Throughput curves for the two methods. Therefore, the current results do not establish whether ACache achieves higher throughput or lower memory usage at comparable generation quality.

Questions for authors

The results in Lines 393–395 are presented only as aggregate numbers, without clearly specifying the datasets, models, and other experimental settings included. Is it appropriate to average accuracy across such heterogeneous tasks? Does excluding BABILong Suffix affect the conclusions?

Reviewer_yw7mOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ACache, a cross-request KV cache reuse mechanism for diffusion large language models (DLLMs). To address the challenges introduced by bidirectional attention, ACache identifies a small subset of important affix tokens, termed Anchor Tokens, using mask-to-affix attention scores. It selectively recomputes their KV states while reusing the remaining shared cache across requests.

The authors further develop a system supporting shared cache reuse at different positions, including prefixes, infixes, and suffixes. Experiments demonstrate that ACache can substantially reduce recomputation overhead and memory consumption while largely preserving generation quality.

Critical strengths
  1. Comprehensive and carefully designed experimental evaluation.

The experimental evaluation is a major strength of this work. The authors evaluate ACache on two diffusion language models (LLaDA and Dream), three benchmarks (GSM8K, MBPP, and BABILong), and two existing inference frameworks (dKV-Cache and Fast-dLLM). The experiments cover different affix positions, anchor ratios, few-shot settings, and batch sizes, providing convincing evidence of the method's effectiveness across diverse configurations.

  1. Meaningful system-level efficiency improvements.

Beyond accuracy evaluations, the paper presents an actual inference system implementation and reports improvements in recomputation latency, end-to-end throughput, and KV cache memory usage. The reported gains of up to 1.71× throughput and 45.8% memory reduction demonstrate the practical value of the proposed approach.

  1. Useful ablation studies and comparisons with related methods.

The paper includes comparisons with CacheBlend, ProphetKV, and BiCache, as well as analyses of anchor stability, selection overhead, and alternative token selection strategies. These experiments help validate the design choices and strengthen the empirical findings.

Critical weaknesses
  1. Limited explanation of the different accuracy trends across tasks and affix positions.

Figure 5 shows that the sensitivity to the Anchor Ratio varies considerably across datasets and affix positions. While the paper discusses the unusual behavior of BABILong under suffix placement, the reasons for the broader differences between prefix, infix, and suffix settings are not fully explained. A more detailed discussion of how task characteristics and affix placement affect the effectiveness of Anchor Selection would help readers better understand when and why ACache performs particularly well.

Questions for authors

Could the authors provide more intuition about why different tasks and affix positions exhibit different sensitivities to the Anchor Ratio, particularly the relatively large differences observed on MBPP?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.