acceptodds
Under review as a conference paper at ICLR 2027

EXACT WARM-STATE REUSE FOR HYBRID DIFFUSION LANGUAGE MODELS: CONTRACTS, SPEEDUPS, AND FAILURE BOUNDARIES

Abstract

Hybrid diffusion language models repeatedly revise a small set of tokens while most context is already stable, yet conventional inference can replay the entire sequence at every denoising step. Reuse is harder than causal prefix caching because a hybrid model combines paged attention state with recurrent Gated Delta Network (GDN) state, noncausal masks, absolute positions, and edit-dependent validity. We formulate a fail closed incremental execution contract over regions. A cache entry binds physical key-value pages and layer-local recurrent frontiers to token, position, model, adapter, mask-contract, and parent version identities. Our accepted GPU fast path is deliberately narrower than the general contract: a stable contiguous prefix and an active contiguous suffix, for which the attention/MLP active rows and the ordered GDN continuation are the same suffix. On one frozen HybridDiffusion- 2B checkpoint and one NVIDIA A30, warm reuse reduces measured serving-route CUDA latency from 1,055.27 to 137.39 ms (7.68×) for a 2,048-token prefix, 64-token suffix, and four diffusion steps, with bit-identical output hashes and 18.2% lower route-local peak memory. Across 36 controlled shapes, median warm speedup is 3.03×; 100 randomized region-contract cases pass their recorded correctness gates. We also report contrary evidence: natural generation is 11.99% slower, a KV-only route fails physical-page validation, and a four-region GPU route faults. We therefore claim an exact contiguous warm-state mechanism and a measured architecture envelope, not general fragmented execution, universal serving speedup, or cross-model transfer.

Primary area:
generative models
Keywords:
Diffusion Language Models, Autoregressive Language Models, Hybrid Generation, Adaptive Inference, KV Cache Reuse, Computation Reuse, Region-Level Refinement, Dependency-Aware Caching, Gated Delta Networks, State Invalidation, Learned Routing, Efficient LLM Inference, Latency–Quality Trade-off, Parallel Decoding, Test-Time Compute.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rNhdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a systematic study on accelerating the inference for Hybrid Diffusion Language Models, where consists of both full attention and GDNs. Authors propose Exact Warm-State Reuse to cache both KV cache and GDN memory states. Exact Warm-State Reuse decides what regions need to be cached and only re-computes changed suffix. The experimental results show median speedup to 3.03x.

Critical strengths

Motivation

  • The paper focuses on a practical problem for hybrid models, where KV Cache and hidden states in the GDN should be jointly considered.

Method

  • The paper proposes a well-designed mechanism, Exact Region-Based State Reuse, as a solution.
Critical weaknesses
  • Exactness under the native attention mask is confusing. It is the proposed Region-DAG mask provides the exact equivalence, instead of the native bidirectional attention from the model. The hidden states of stable tokens may change after each denoising step. Can you demonstrate whether the proposed caching mechanism saves the native inference dynamics from its native bidirectional attention or not?
Questions for authors

N/A.

1 comment
AuthorsOpenReview ↗

Native inference fidelity and the localized row-geometry discrepancy

Thank you for identifying the central ambiguity. We agree that matching a declared attention mask and controlled prediction hashes does not, by itself, establish preservation of native hybrid inference dynamics.

  1. Scope of exactness

Our supported claim is bitwise cached execution relative to a declared, shape-matched canonical execution contract. It is not a general claim of bitwise equivalence to native generation under every execution schedule.

The audited native configuration uses causal prompt prefill and a mixed clean/noisy self-speculative block. Matching attention edges is nevertheless insufficient to match the complete hybrid computation: the separate fixed-state native/declared diagnostic does not match across its four tested steps. GDN policy and finite-precision execution schedule must also be considered.

  1. Additional real-serving diagnostic

We separated four serving paths:

A: pristine native self-speculative generation. B: ordinary exact handoff without cross-edit transfer. C: fresh segmented prefix/suffix reconstruction. D: cross-edit joint-state restoration.

In the bounded diagnostic, A and B match on all eight observed outputs and all 563 aligned sampled tensor comparisons. C and D match on all eight outputs and all 696 aligned sampled comparisons. Native and segmented outputs differ. Thus, the discrepancy can arise without cross-edit restoration.

  1. Localization of the first observed divergence

The latest raw-tensor experiment shows bitwise-identical embedding, normalization, and projection inputs, with identical recorded projection weights. All six own-shape projection replay checks reproduce their captures bitwise.

Changing projection execution from 946 rows to 11 rows reproduces the QKVZ discrepancy: 61 of 90,112 suffix values differ, with maximum absolute error 0.015625. Diagnostic 64-row padding reproduces the native projection output for this case.

This supports a row-geometry numerical explanation for the first observed causal-prefill divergence. It is not a full-model repair, does not explain every older mixed-GDN discrepancy, and does not establish preservation of denoising trajectories.

  1. Remaining limitation and revision

The sequential-edit benchmark records genuine restoration and invalidation, but native and restored-route final outputs match in 0/360 paired requests. We will retain this negative result and will not claim native-equivalent editing acceleration.

We will revise the manuscript to distinguish dependency eligibility, bitwise reuse against the canonical reference, and fidelity to native finite-precision generation. The new diagnostic explains the first observed divergence while leaving the broader native-equivalence limitation explicit. We will also avoid treating equality of completed answer tokens as proof that internal states or denoising trajectories are identical.

Reviewer_f6hPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cache reuse in hybrid diffusion language models. Since KV caching alone cannot restore GDN recurrent states, the authors cache both KV and GDN states and check positions, versions, and physical GPU page mappings. A Region-DAG manages dependencies and cache invalidation, but the validated GPU fast path is limited to a stable contiguous prefix and an active contiguous suffix. Controlled experiments show a median speedup of 3.03×, with some cases reaching 7.68×. Natural generation produces identical outputs but runs 11.99% slower.

Critical strengths

The authors cache both KV and GDN states and check positions, dependency versions, and physical GPU page mappings to reduce the risk of errors from stale cache states. The experiments show that avoiding repeated computation reduces latency and GPU memory use. The authors also report slower natural generation and failures on some GPU paths, showing the method's limitations.

Critical weaknesses
  1. Exact reuse requires stable queries not to attend to active tokens, but native bidirectional attention may not satisfy this restriction. It remains unclear which practical scenarios meet this condition. The 15-prompt validation is limited, and the paper does not specify what the output hashes compare.

  2. The paper shows speedups from KV/GDN caching but does not explain what dependency management and runtime validation add beyond basic prefix caching. Natural generation was 11.99% slower, leaving the practical benefits unclear.

Questions for authors
  1. Which practical generation or editing scenarios satisfy the restriction that stable queries cannot attend to active tokens? Is the model's native attention behavior preserved? What do the output hashes compare?

  2. What do dependency management and runtime validation add beyond basic KV/GDN prefix caching? Can the authors provide a concrete example and timings that include cache construction and reuse?

  3. Figure 1 lists region-four-mid-d4 as accepted, but Section 6 reports a failure of the four-fragment GPU path. How do their configurations and acceptance criteria differ?

2 comments
AuthorsOpenReview ↗

Applicability, hash semantics, validation value, and Figure 1 provenance

Thank you for requesting clearer applicability conditions, a precise definition of exactness, and an explanation of the fragmented-case inconsistency.

  1. Applicable scenarios and native behavior

Reuse requires independence under the actual execution contract: retained prefix state must not depend on the edited suffix and must have been constructed under compatible attention and recurrent policies.

Eligible examples include causal-prefix continuation and suffix edits after an eligible retained boundary. Unchanged token text alone is insufficient; arbitrary bidirectional infilling is not covered.

Sampled native causal-prefill perturbations preserve the observed prefix hidden-state and KV hashes after suffix changes. This supports eligibility in that tested regime, not arbitrary masks, edits, or recurrent boundaries. The audited native configuration uses causal prompt prefill and a seven-row self-speculative block with four causal clean rows and three noisy rows with broader block visibility. We will specify the mask and GDN policy explicitly.

  1. What the hashes compare

The controlled timing benchmark hashes the ordered top-1 predicted token IDs at logical active positions across diffusion steps. Each integer is encoded as signed 64-bit little-endian bytes before SHA-256 hashing.

This does not hash complete logits, all recurrent states, or a completed generated answer. We will distinguish:

(a) Controlled timing acceptance: active-position top-1 predictions across steps. (b) Fixed-state tests: bitwise logits, hidden-state, and GDN-state comparisons against the shape-matched segmented reference. (c) Complete-generation comparisons: actual generated token IDs and decoded text.

All 650 measured prediction hashes in the new controlled benchmark were independently recomputed. Fixed-state agreement is separate from a claim of preserved native denoising trajectories.

  1. What validation adds beyond basic joint caching

The tests include a concrete stale-state example: prefix tokens remain unchanged while positions change. A token-only identity remains unchanged, whereas the exact cache rejects reuse. Further cache-API tests reject incompatible parent versions, mask contracts, model/adapter identities, recycled request slots, and capacity eviction.

These are CPU contract tests, not concurrent GPU allocator-stress results. A correct simple joint cache should reject incompatible identities too; we do not claim this capability is exclusive to ExactRegion.

For the 2,048-prefix/64-active/four-step case, separately measured frontier establishment costs 231.93 ms, cold handoff including establishment costs 370.73 ms, and warm execution costs 139.76 ms. The simple joint cache costs 140.06 ms and nearly ties our method. Full replay and conservative KV-only reconstruction cost 1,063.66 and 1,070.61 ms, respectively.

Thus, the extra machinery is supported as an explicit safety and auditability contract, not a demonstrated additional throughput benefit. We have not isolated a universal validation-overhead number. We will distinguish route timing, required runtime checks, post-route acceptance, and whole-harness time.

  1. Figure 1 versus the failed four-fragment case

The local Figure 1 latency PDF is byte-identical to the figure in the original results archive. Its region-four-mid-d4 case uses revision c57970d, sequence length 1,536, and four 32-token active spans (128 logical active tokens). It reproduces median full/warm latencies of 1,050.50/893.59 ms with passing recorded checks.

The failed comparison-four-regions-d4 case uses revision 875c031 and four 16-token spans (64 active tokens), and faults before producing accepted cached measurements. These are different workloads and revisions, not contradictory outcomes for one identical configuration. Their differences do not establish the cause of the CUDA fault.

Crucially, the successful characterization performs dense replay after the earliest frontier: it submits 5,120 query positions over four steps, not merely 128 x 4 logical active positions. It therefore does not establish independent reuse of downstream stable regions or reliable sparse fragmented execution.

We will correct the broad caption, separate this dense-frontier characterization from accepted contiguous-serving claims, report actual executed work, and retain the fragmented-route failure. We will also preserve the distinction between controlled-reference exactness and native-generation fidelity; the new sequential-edit experiment does not establish native-equivalent acceleration.

Thank you for the detailed response. The clarifications have helped me better understand the method and its limitations. I consider the clarified scope and experimental evidence sufficient to raise my rating from 2 to 3.

Reviewer_i5dMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper caches attention KV pages and Gated DeltaNet state across denoising steps of a hybrid diffusion LM, using a region contract and a fail-closed cache key so that outputs stay bit-identical. Only the contiguous prefix or suffix case is accepted on GPU. On one 2B model, one A30 and batch size 1, warm reuse is 7.68x faster than full replay in the main case (median 3.03x over 36 cases), but 11.99% slower than the model's native self-speculative route.

Critical strengths

The paper keeps its negative results (Table 7). Exactness is checked with output hashes, and the observation that BF16 kernels vary with row shape (L142-144) is useful for anyone building exact caches.

Critical weaknesses
  1. What's validated is exact prefix reuse: keep the prefix KV pages and the GDN state at the boundary, then recompute the suffix. Both parts have close prior art (prefix caching in BD3-LM and Fast-dLLM, recurrent-state checkpointing in Marconi and Sparse Prefix Caching). Because GDN is recurrent in textual order (L123), the earliest edit sets the replay frontier (L285), so the general Region-DAG contract reduces to prefix reuse here, and the four-fragment GPU route fails (Table 7).
  2. The speedups are all measured against full replay, which recomputes the whole prefix at every step (L209-210), so much of the gain is built in. The KV-only baseline failed validation (L315-316), and the one comparison with the native route is negative (L310-312).
  3. The reference uses the same Region-DAG mask as the cached path (L320-323), and stable queries can't attend to active keys (L111). If the native mask lets the prefix see the suffix, the outputs are exact with respect to a modified model, not the original one.
  4. The up to 14.92x (L64) comes from a context-scaling sweep (Table 4) that isn't among the accepted evidence collections in Table 2, so I can't tell whether it passed the exactness checks. Appendices A, D, E and I are cited but missing from the PDF, so I couldn't check the proof. The writing reads like an internal audit log, and Fig. 1 is unreadable at print size.
Questions for authors
  1. Is there any case where a stable region after an edit is reused exactly?
  2. What is the checkpoint's native attention mask, and is the checkpoint public?
  3. How does the method compare with a working KV-only + GDN-replay baseline, and with the native route on realistic editing workloads?
1 comment
AuthorsOpenReview ↗

Additional baselines, native-editing evaluation, and verified evidence scope

Thank you for distinguishing our general contract from the execution actually validated. The additional experiments clarify both the benefits and the remaining limitations.

  1. Scope and prior work

The validated fast path reuses a contiguous prefix's KV pages and GDN boundary states, then recomputes the suffix. We have not established independent exact reuse of a stable region downstream of an edited span. Logical independence in the attention DAG does not eliminate ordered recurrent dependencies.

We agree that joint prefix caching and recurrent checkpointing have close precedents in Marconi, Sparse Prefix Caching, Block Diffusion, and Fast-dLLM. We will strengthen this positioning and avoid presenting joint prefix caching itself as novel. Our supported contribution is the explicit validity contract, its serving integration, and the empirical characterization of correctness and failure boundaries in this hybrid-diffusion stack.

  1. Working baselines

For a 2,048-token prefix, 64-token active suffix, and four diffusion steps, the new median CUDA route latencies over ten measured repetitions are:

Canonical segmented full replay: 1,063.66 ms. KV-only reuse with conservative GDN reconstruction: 1,070.61 ms. Simple joint KV/GDN prefix cache: 140.06 ms. ExactRegion warm execution: 139.76 ms. Cold handoff, including frontier establishment: 370.73 ms.

Recorded active-position prediction hashes agree across all five routes. Separately measured frontier establishment costs 231.93 ms and is excluded from steady-state warm latency.

The simple joint cache nearly ties ExactRegion: approximately 0.21% difference in this case. We therefore attribute the large saving over full replay primarily to joint KV/GDN reuse, not a demonstrated additional latency benefit from DAG management. These adapted baselines share scheduler/mask scaffolding; they are not direct executions of Marconi, vLLM, or upstream SGLang. Conservative reconstruction is not claimed to be the fastest possible KV-only implementation.

  1. Native mask and public checkpoint

The audited self-speculative configuration uses causal prompt prefill. Its seven-row decoding block has four causal clean rows followed by three noisy rows that can attend throughout the block. It is not unrestricted whole-sequence bidirectional attention. We will document the GDN mixed-mode policy too, since matching attention edges alone does not establish matching hybrid computation.

The checkpoint is publicly available as yuchen-zhu-zyc/HybridDiffusion-2B. All five required frozen files match public revision 04e16a066c17a512f9a567b01362d6ab811bcf4b, including the weight-shard SHA-256.

  1. Native generation versus sequential editing

Ordinary exact handoff and native generation match on 200/200 completed output pairs, comprising 20 constructed prompts and ten repetitions. This is not a cross-edit restoration test or proof of matching internal trajectories.

We separately compared native generation and the cross-edit bridge over 20 constructed document sequences, five edits plus initial generation, and three repetitions. The bridge records 240 eligible restores, 60 initial builds, and 60 prefix-change rebuilds. However, final outputs match in 0/360 pairs. Median cumulative sequence latency is 10,262.66 ms native versus 10,437.62 ms with the bridge.

We therefore report different-output route timings, not native-equivalent acceleration. We will label these sequences as constructed rather than an independently sourced realistic workload.

  1. Sweep acceptance and presentation

We re-audited all 16 original context-scaling full/warm comparisons. Each has ten samples per route, complete repetition IDs, matching recomputed prediction hashes, and passing recorded position, work-accounting, fallback/recovery, synchronization, and canonical-reuse checks.

The maximum is 14.92444x, with a 4,055-token prefix, 41 active tokens, and four steps. We will add this collection explicitly to the evidence table. Its acceptance is relative to the controlled reference, not native-generation equivalence.

We will enlarge Figure 1, clarify characterization versus accepted serving results, and verify that the revised uploaded PDF includes every cited appendix and correct cross-references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.