acceptodds
Under review as a conference paper at ICLR 2027

REPRISE: Efficient Long-Context Diffusion Inference via Distillation through Reused KV Caches

Abstract

Bidirectional diffusion language models repeatedly refresh key–value (KV) caches over long prompts during generation, incurring substantial computational cost. Sparse attention and cache reuse reduce this cost but alter the context states used for prediction, which can reduce generation quality. To address these limitations, we propose REPRISE, which adapts models to sparse attention and cache reuse through dense-model distillation and selective replay. During training, the student makes predictions using sparse attention and reused caches. A frozen dense teacher supervises these predictions based on the student’s current token sequence. Selective replay rebuilds the required cache computations so that later prediction losses can train cache construction without retaining the full generation graph. During inference, REPRISE refreshes prompt states less often than generation states and uses sparse attention for prompt refreshes. Compared with the widely used Prefix Cache and Dual Cache, REPRISE improves average scores by 8.72 and 7.67 points, respectively, while reducing time to first token (TTFT) by 48% and time per output token (TPOT) by 80–82% on average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, long-context inference, sparse attention, KV caching, knowledge distillation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WsBVOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes REPRISE, a training-based framework for accelerating long-context inference in bidirectional diffusion language models. During inference, REPRISE combines sparse attention with separate KV-cache refresh schedules for prompt and generation tokens. During training, a frozen dense teacher supervises a student operating under sparse attention and reused KV caches. The proposed selective cache replay reconstructs the computational dependencies of historical KV caches, allowing subsequent prediction losses to update earlier cache construction without backpropagating through the entire generation trajectory. Experiments on RULER and LongBench demonstrate improved inference efficiency and generation quality compared with Prefix Cache and Dual Cache

Critical strengths
  1. Significant inference efficiency improvements. REPRISE achieves substantial speedups over conventional KV-caching methods, particularly in long-context settings, while maintaining or improving generation quality on RULER.

  2. An interesting approach to training under approximate inference. Rather than relying exclusively on training-free approximations, REPRISE explicitly adapts the model to sparse attention and stale KV caches. Its selective replay mechanism provides an efficient way to preserve gradients through historical cache construction.

  3. Detailed experimental analysis. The paper includes controlled training ablations, multiple random seeds, and statistical analyses that help distinguish the effects of sparse distillation, cache reuse, and historical cache gradients.

Critical weaknesses
  1. Poor organization and unclear presentation. The paper is unnecessarily difficult to follow due to its organization and uneven level of detail. Section 4.1 revisits observations already explored in prior work, such as prompt KV stability in dLLM-Cache [1], while presenting the speedup of sparse prompt refreshes before explaining the actual sparse attention strategy. Equation (2) introduces the function ℓ without explicitly defining it. Furthermore, Section 4.2 presents ablation results intended to demonstrate REPRISE's effectiveness before the method itself is introduced in Section 5. These organizational issues substantially hinder readability and obscure the paper's contributions.

  2. Limited methodological novelty. The motivation and training mechanism closely resemble Self Gradient Forcing (SGF) [2], which already identifies the missing gradient path through historical KV caches and addresses it through no-gradient rollout followed by differentiable cache reconstruction. Sparse-aware fine-tuning has also been explored in MAGE v1 [3]. Moreover, the separation of prompt and generation cache updates has been investigated by dLLM-Cache [1], while the hierarchical management of historical and active-block caches resembles Fast-dLLM v2 [4], albeit under a different attention architecture. Overall, REPRISE appears primarily to combine and adapt existing techniques rather than introduce a fundamentally new methodology.

  3. Limited practical relevance to emerging dLLM architectures. Recent prominent dLLMs, such as Fast-dLLM v2 [4] and LLaDA2.0 [5], increasingly adopt block-diffusion architectures, where block-causal attention enables exact KV-cache reuse across completed blocks. This largely eliminates the stale prompt-cache problem that motivates REPRISE. Consequently, the practical value of its cache-adaptation mechanism for newer dLLMs remains unclear.

  4. Insufficient comparisons with long-context-specific methods. Despite targeting efficient long-context dLLM inference, the main experiments primarily compare REPRISE against conventional caching methods. More relevant baselines include Prefilling-dLLM [6], which addresses redundant long-prompt computation, Focus-dLLM [7]. Without comparisons against these methods under comparable long-context workloads, it is difficult to assess REPRISE's competitiveness in both inference efficiency and generation quality.

Reference

[1] Liu et al. dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching. ICML 2026. https://arxiv.org/abs/2506.06295

[2] Zhuang et al. Self Gradient Forcing: Native Long Video Extrapolation. arXiv 2026. https://arxiv.org/abs/2607.20368

[3] Kwon et al. MAGE: All-[MASK] Block Already Knows Where to Look in Diffusion LLM. arXiv 2026, v1. https://arxiv.org/abs/2602.14209v1

[4] Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026. https://arxiv.org/abs/2509.26328

[5] Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv 2025. https://arxiv.org/abs/2512.15745

[6] Xiong et al. Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models. arXiv 2026. https://arxiv.org/abs/2606.10537

[7] Long et al. Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing. ACL 2026. https://aclanthology.org/2026.acl-long.556/

Questions for authors

How does the proposed sparse attention strategy differ from SparseD? Section 5.1 introduces a pooled Q/K-based block selection strategy that appears conceptually similar to SparseD. What are the specific methodological advantages of this design, and why was SparseD not directly adopted?

11 comments
AuthorsOpenReview ↗

# Response to Reviewer WsBV

Thank you for your detailed review and for recognizing the inference benefits, the training objective, and the experimental analysis. We have revised the organization, clarified the methodological contribution and architectural scope, and added the requested comparisons. We address each weakness and the question below.

W1: Organization and clarity of presentation

Thank you for identifying the specific presentation issues. We have revised Section 4 to introduce the execution mismatch and training cost before the method, placed trained-model comparisons after the method, and supplied the missing definitions.

1. Relating prompt-KV stability to prior work — Section 4.1

Section 4.1 now explicitly credits dLLM-Cache for using the temporal stability of prompt states to reduce cache refreshes. Our measurements examine this observation under the backbone and evaluation settings used in this paper and motivate different refresh frequencies for the prompt and generation regions. The revised text makes this relationship to prior work explicit.

2. Explaining the sparse strategy before its speedup — Section 4.1; full specification in Section 5.1 and Appendix A.3

Before reporting latency, Section 4.1 now explains the strategy used in the measurement: 128-token prompt blocks, pooled Q/K scores for key-block selection, and a nominal 25% prompt-key-block budget. Interactions involving generation tokens and the incomplete prompt-tail block remain dense. Readers can therefore understand the measured strategy before interpreting its speedup, with the exact rule specified in Section 5.1 and Appendix A.3.

3. Defining the loss before Equation (2) — Section 4.2

We now define ℓ(φ, C) as the distillation loss against a fresh dense teacher, computed by subsequent operations using parameters φ and the supplied prompt cache C. The later prediction loss is written as L(φ) = ℓ(φ, C(φ)). Equation (2) then explains the gradient paths through the current prediction computation and the earlier cache construction. The full KL objective appears in Section 5.2, Equation (4).

4. Placing training ablations after the method and revising the motivation — Section 4.2, Figure 2(c–d), and Section 6.3, Figure 4

The trained-model comparisons are now presented in Section 6.3 and Figure 4. Section 4.2 and Figure 2(c–d) instead contain two diagnostics that motivate the method:

  • Figure 2(c) compares prediction discrepancies under different sparse/cache policies against a fresh dense teacher, without updating pretrained weights. This motivates adaptation under the actual cache-reuse states.
  • Figure 2(d) measures the time and memory required for full-history reconstruction and backward computation. This motivates reducing the training cost of retaining cache-construction gradients.

These diagnostics lead into cache-aware training and selective replay in Section 5. Their effectiveness is evaluated afterward in Section 6.3.


W2: Methodological contribution relative to SGF and prior work

Thank you for pointing out these connections. We have clarified the relationship between the existing components and our contribution in Sections 2 and 5.3. REPRISE uses the overwrite structure of generation refreshes in a bidirectional dLLM to identify which cache computations remain dependencies of selected predictions. This lets it skip overwritten generation-cache history while preserving the gradient path from later losses to prompt-cache construction.

1. Relationship to SGF: using refresh boundaries to identify necessary reconstruction — Sections 2 and 5.3

We acknowledge SGF's use of no-gradient trajectory collection and differentiable context reconstruction to train historical KV representations through later losses. Section 2 explicitly discusses this connection.

REPRISE uses the particular dependencies of its cache policy: at a generation-refresh boundary, every layer recomputes all generation KV entries while continuing to reuse prompt KV. On a fixed saved token trajectory, predictions after that boundary depend on prompt-cache construction, while the overwritten generation-KV computations can be skipped.

For example, a probe in the fourth generation block requires reconstruction of the initial prompt cache, the generation cache at the fourth-block boundary, and the subsequent active-block computation. The first three blocks' generation forwards need not be replayed, while the later loss still differentiates through the reconstructed prompt cache. Section 5.3 now states this overwrite relationship and the basis for retaining or skipping each computation.

2. Relationship to earlier cache policies — Sections 2 and 5.1

We explicitly cite dLLM-Cache's separate update strategies for prompt and generation states and Fast-dLLM v2's block/sub-block cache management. REPRISE retains global bidirectional attention and jointly considers sparse computation, approximate cache reuse, and training under this architecture.

The refresh policy determines both which states are reused during inference and which historical computations affect the selected training losses. Section 5.1 specifies this deployment policy, and Section 5.3 constructs differentiable reconstruction from its dependencies so that training directly matches the sparse cached states used at deployment.

3. Relationship to sparse-aware fine-tuning: testing reused-state training and cache-construction gradients — Section 6.3, Figure 4, and Appendix D

MAGE v1 studies lightweight fine-tuning for sparse attention, and is explicitly cited in Section 2. To examine REPRISE's training components, we compare fresh sparse distillation, detached prompt-cache training, and REPRISE under the same deployment policy, training data, adaptation rank, and update budget. Results on 416 RULER 64K inputs, averaged over three training seeds, are:

Training methodMean scoreREPRISE score gainPaired 95% confidence interval
Fresh sparse distillation78.24+1.38[0.40, 2.43]
Detached prompt-cache training76.67+2.95[1.74, 4.23]
REPRISE79.62——

Both gains hold for every matched training seed. They support training under reused states and allowing later losses to differentiate through prompt-cache construction, respectively.

4. Direct gradient and resource measurements for selective replay — Section 6.3 and Appendix B.5, Table 9

We added a direct comparison with layerwise activation-checkpointed full-history reconstruction, using identical saved trajectories, token-transfer decisions, attention selections, and selected losses. Across 14 jointly completed configurations and three repetitions each, all 42 paired adapter-gradient comparisons have maximum absolute difference 0, relative L2 difference 0, and cosine similarity 1.

One configuration in Table 9 uses a 512-token prompt, a 512-token generation canvas, and probe position 96, measured on the same H100:

Reconstruction methodUpdate time (s)Peak GPU memory (GiB)
Full history with activation checkpointing42.5864.34
Selective replay3.7317.90

For later-block probes with 512-token prompts, selective replay uses 8.7%–17.6% of the reference update time and 22.4%–46.6% of its peak memory. With 4,096-token prompts, it also completes the eight later-position configurations where the reference runs out of memory. Gradient agreement concerns identical selected losses on fixed discrete trajectories; these measurements directly establish the benefit of reducing reconstruction through refresh-boundary dependencies.


W3: Practical relevance relative to block diffusion

Thank you for pointing out how attention architecture affects cache reuse. We agree that the block-causal structures in Fast-dLLM v2 and LLaDA2.0 allow exact KV reuse across completed blocks. We have clarified REPRISE's architectural scope and added a discussion of its relationship to block diffusion in prefill optimization.

1. REPRISE targets dLLMs that retain global bidirectional attention — Sections 1, 2, and 3

Sections 1 and 2 state this scope and discuss block-causal models including Fast-dLLM v2 and LLaDA2.0. Section 3 explains that, under global bidirectional attention, changing generated content can alter deeper-layer prompt representations even when the prompt tokens themselves remain fixed. Reusing earlier prompt KV is therefore approximate.

REPRISE trains under this deployment state to adapt the model to sparse attention and cache reuse. Our objective is to reduce quality loss while accelerating inference, and to improve quality in settings where adaptation permits it.

2. Retaining cross-block bidirectional context has practical task value — Section 1

Section 1 now cites Set Diffusion and Speculative Correction. Set Diffusion demonstrates benefits from more flexible cross-block context use relative to block diffusion in text infilling. Speculative Correction's GSM8K ablation shows a benefit from global answer refinement relative to local refinement. These results support the value of cross-block context in the corresponding tasks and motivate efficient execution of models that retain this capability.

3. Sparse prefill and exact cache reuse are potentially complementary — Sections 5.1 and 5.4

We have added this discussion in Section 5.4. For a new prompt without a reusable cache, block diffusion still requires initial prefill to construct prompt KV. Exact cache reuse reduces repeated computation after construction, while sparse prefill can reduce attention computation during initial construction itself.

Section 5.1 specifies that REPRISE applies sparse prompt attention from initialization, without first running dense attention to collect a sparsity pattern. It therefore reduces attention computation in both the first cache construction and subsequent refreshes.

This sparse-prefill design offers a direction for extension to block diffusion: adapt sparse selection and training to the block-causal attention structure and cache dependencies, combining cheaper initial construction with subsequent exact reuse. Section 5.4 presents this as potential complementarity; the measured results in this paper concern UltraLLaDA-8B with global bidirectional attention.


Reviewer_ttJ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

REPRISE adapts a bidirectional diffusion language model to inference with sparse attention and reused KV caches. It refreshes prompt states less frequently than generation states and applies block-sparse attention during prompt refreshes. During training, a frozen dense teacher supervises predictions at student-generated token states using full-vocabulary forward KL divergence. The central training mechanism is selective cache replay: the method reconstructs the initial prompt cache and selected generation-boundary and active-block computations with gradients enabled. This allows later prediction losses to train earlier cache construction without retaining the full rollout graph. The evaluation uses UltraLLaDA-8B on RULER and LongBench, with three adaptation seeds and controls for fresh sparse distillation and detached prompt caches. RULER results support the benefits of cached-state training and prompt-cache gradients. Complete-generation experiments with parallel decoding report speedups over Dual Cache of 5.04× on 26 RULER inputs and 1.55× on 24 LongBench inputs. Quality improvements on LongBench V1 remain statistically uncertain, while LongBench V2 gains mainly reflect improved output validity.

Critical strengths

S1. The training design addresses a concrete mismatch between training and cached inference, with meaningful controlled evidence. REPRISE supervises predictions made using sparse, reused states and allows their losses to update earlier cache construction. Under a shared deployment policy, it improves RULER 64K scores over fresh sparse distillation by 1.38 points and over detached prompt-cache training by 2.95 points, with paired confidence intervals excluding zero. The three-seed comparisons and explicit gradient-intervention checks provide useful evidence that cached-state training and prompt-cache gradients contribute beyond ordinary sparse distillation. S2. The paper demonstrates practical inference benefits and reports important limitations transparently. Under shared confidence-based parallel decoding, REPRISE achieves a 5.04× complete-generation speedup over Dual Cache on RULER 32K, alongside an 8.59-point score improvement. The LongBench experiment also shows a measured latency benefit. The paper distinguishes fixed-step probes from complete generation, identifies differences in the NoCache references, and explicitly acknowledges uncertain LongBench quality gains and the output-validity explanation for LongBench V2. These disclosures make the evidence easier to assess.

Critical weaknesses

W1. Evidence for general quality improvements is substantially narrower than the aggregate benchmark gains suggest. The strongest evidence comes from RULER. All main LongBench V1 source-paired score-difference intervals include zero, and the LongBench training ablations do not establish an advantage over fresh sparse distillation or detached-cache training. LongBench V2 improvements mainly occur on examples with unparseable baseline outputs; even NoCache frequently produces repeated newlines. These results demonstrate improved behavior in the evaluated setup but provide limited evidence of stronger long-context reasoning or general natural-task quality. Evaluation on one backbone and small benchmark subsets further limits the scope of the conclusions. W2. The distinct methodological and resource-efficiency contribution of selective replay needs more direct evidence. Reconstructing differentiable historical KV states without backpropagating through the full rollout is related to Self Gradient Forcing, while sparse-aware fine-tuning has also been explored in MAGE. REPRISE provides a plausible specialization to bidirectional language-model caches and refresh boundaries, and acknowledges these connections. However, the experiments primarily establish the value of cache-aware supervision and gradient attachment. They do not directly quantify the memory/computation benefit of selective replay against a checkpointed full-rollout reference, or establish gradient agreement with that reference for identical selected losses and a fixed trajectory. Reported training time and replay-logit agreement are useful but do not fully characterize this central contribution.

Questions for authors

Q1 — Selective replay correctness and efficiency. On a tractable fixed trajectory, can you compare selective replay against retaining or checkpointing the complete cache-update graph, using identical selected loss probes? Please report gradient agreement, peak GPU memory, and update time. How do these quantities scale with generation length and the selected probe’s position? Q2 — Interpretation of natural-task improvements. Can you provide a matched diagnostic explaining the repeated-newline failures on LongBench V2, including the fresh dense baseline? Does the improvement persist under a validated native or alternative direct-answer decoding setup shared by all methods? This would help distinguish recovery from a decoding/output-validity failure from broader long-context improvements. Q3 — Distinguishing cache adaptation from general distillation. Can you include a same-budget dense-student distillation control, evaluated with both fresh dense inference and the REPRISE deployment policy? The existing fresh sparse control is informative, but this additional comparison would help determine whether part of the gain reflects general behavioral adaptation to dense-teacher targets. Q4 — Latency interpretation. Appendix N defines prefill latency through the first token commitment. Does the reported TTFT correspond to a committed token at any generation position, or to the first available output-prefix token? Also, please distinguish the single-input 1.94× TTFT and 5.09–5.46× TPOT results from averages over requests in the abstract and contribution statements. Q5 — Reproducibility. Will the exact evaluation manifests and exclusion lists be released? Appendix I states that reconstructing the LongBench subset requires archived exclusion IDs that are currently internal. These artifacts are important for reproducing the reported comparisons.

5 comments
AuthorsOpenReview ↗

# Response to Reviewer ttJ5

Thank you for your positive assessment of the training design, inference efficiency, and experimental analysis, and for your specific suggestions on selective replay, natural-language evaluation, and reproducibility. We have added the requested measurements and updated the main text and appendices. Our responses follow.

W1 & Q2: Natural-language quality and the LongBench V2 evaluation

REPRISE aims to accelerate long-context inference while reducing the quality loss introduced by sparse attention and cache reuse, and improving quality where possible. Sections 1 and 6.2 explain the results in terms of this quality–efficiency objective.

1. Re-evaluating V2 under a validated shared direct-answer protocol — Sections 6.1–6.2, Table 1, and Appendix L, Table 23

Under the previous long-form answer configuration, some outputs consisted of repeated newlines, including those from fresh dense NoCache. We therefore revisited the shared prompt and decoding settings. Keeping the original questions and context-truncation rules, we standardized the final answer prompt to Answer:, used FP16 computation and KV, zero temperature, a four-token output budget, and the same leading-option extraction rule for every method. We checked the short-answer decoding path against the backbone's native implementation on validation inputs.

We repeated the evaluation on the original 104 questions at all three input caps, including all three REPRISE training seeds. Every compared method now produces a valid option on every question. The following results report accuracy in percent, with the mean and sample standard deviation across the three REPRISE seeds.

Input capNoCachePrefixDualREPRISE
32K29.8132.6932.6934.62 ± 0.96
64K29.8135.5835.5835.90 ± 1.47
128K31.7335.5835.5834.94 ± 1.11

REPRISE's mean accuracy exceeds NoCache at all three input caps and exceeds Prefix and Dual at 32K and 64K. At 128K, it is 0.64 percentage points below the two cache baselines. Section 6.2 and Table 1 report these results; Appendix L, Table 23 reports all individual training seeds.

2. Reporting paired statistics for V1 and the training ablations — Sections 6.2–6.3, Appendices L.1 and O

In the main V1 quality comparison, REPRISE's mean score exceeds Prefix and Dual at all three input caps and remains close to fresh dense NoCache. All nine source-paired score differences and confidence intervals are retained in Appendix L.1, Table 24.

For the training ablation on 256 natural-language inputs, Section 6.3 and Appendix O, Tables 30–32, report individual-seed results, paired intervals, and the task-level breakdown under the same deployment policy:

Training methodMean scoreREPRISE score gainPaired 95% confidence interval
Fresh sparse distillation39.4120+0.9008[−0.3604, 2.1836]
Detached prompt-cache training39.9326+0.3801[−1.3408, 2.0632]
REPRISE40.3128——

Section 6.2 and Appendix P, Table 33, also report a 1.55× complete-generation speedup over Dual on the 24-input V1 comparison, with mean scores of 41.77 and 40.46, respectively. These results evaluate quality preservation and improvement during accelerated inference. Section 6.1 specifies the UltraLLaDA-8B backbone, evaluation subsets, and checkpoints to which the results apply.


W2 & Q1: Selective replay, gradient agreement, and training cost

We have added a direct comparison with activation-checkpointed full-history reconstruction, using identical fixed trajectories and selected losses to measure gradient agreement, update time, and peak GPU memory. The method is explained in Section 5.3, the results are summarized in Section 6.3, and all configurations appear in Appendix B.5, Table 9.

1. The specific role of selective replay — Sections 2 and 5.3

Section 2 explains the connections to SGF's differentiable cache reconstruction and MAGE's sparse-aware fine-tuning. REPRISE uses the overwrite structure at generation-refresh boundaries to identify which cache computations remain dependencies of selected predictions.

On a fixed saved token trajectory, a generation boundary overwrites all generation KV entries, while the retained prompt cache continues to affect subsequent predictions. Replay therefore rebuilds the corresponding prompt cache, selected generation boundary, and subsequent active-block computation. It skips the overwritten generation-cache history while retaining the gradient path from later losses to prompt-cache construction.

2. Gradient agreement for identical trajectories and losses — Section 6.3 and Appendix B.5

Both methods use identical saved token states, token-transfer decisions, attention selections, loss probes, initial adapter parameters, and optimizer state. The full-history reference uses layerwise activation checkpointing and releases obsolete graph dependencies after full prompt refreshes.

Across the 14 configurations completed by both methods, with three repetitions each, all 42 paired comparisons give the following measured adapter-gradient agreement:

Gradient metricResult
Maximum absolute difference0
Relative L2 difference0
Cosine similarity1

This verification concerns the same selected losses on fixed discrete trajectories.

3. Scaling with generation length and probe position — Appendix B.5, Table 9

We evaluate 22 configurations spanning prompt lengths of 512 and 4,096 tokens, generation lengths of 128, 256, and 512 tokens, and different probe positions. Measurements use the same H100 GPU, one untimed warmup, and three timed repetitions.

For 512-token prompts and a fixed probe position of 96, the results are:

Generation lengthFull-history update (s)Selective replay update (s)Full-history peak memory (GiB)Selective replay peak memory (GiB)
12841.743.6645.6416.77
25642.293.6951.6817.11
51242.583.7364.3417.90

The first generation block requires similar computation in both methods. For later-block probes with 512-token prompts, selective replay uses 8.7%–17.6% of the reference update time and 22.4%–46.6% of its peak GPU memory.

At generation length 512 and probe position 480, full-history reconstruction takes 34.67 seconds and 77.14 GiB, compared with 3.73 seconds and 17.90 GiB for selective replay. With 4,096-token prompts, the full-history reference runs out of memory in eight later-position configurations, while selective replay completes all 22 configurations.

Update time includes reconstruction, dense-teacher evaluation, backward computation, gradient clipping, and AdamW. The percentages quantify this update stage. Appendix B.5 provides the complete timing boundaries and configurations, including how prompt-refresh positions affect dependency lifetimes and resource usage.


Q3: Distinguishing cache adaptation from general distillation

We have added a dense-student control with the same update budget and evaluated it under both fresh dense inference and the REPRISE deployment policy. Section 6.3, Table 3, reports the results; Appendix B.6 provides training details and paired intervals.

1. Matching training data and update budget — Appendix B.6

The dense student and REPRISE use the same 64 training articles, data order, three training seeds, rank-8 adapters, 64 parameter updates, probe-selection rule, and AdamW hyperparameters. Each dense student collects its own fresh dense trajectory and receives supervision from the frozen teacher at the same saved states.

The matched budget is the number of parameter updates. The three dense-student training runs take 70.51, 71.79, and 71.22 minutes, respectively.

2. Comparing the same deployment policy and reporting the fresh dense reference — Section 6.3, Table 3

The evaluation uses 26 RULER 32K inputs and 256 LongBench V1 inputs. Results are means and sample standard deviations across three training seeds:

TrainingInference policyRULER 32KLongBench V1
Dense-student distillationREPRISE policy70.66 ± 3.4139.29 ± 0.80
REPRISE trainingREPRISE policy81.69 ± 1.9841.61 ± 0.91
Dense-student distillationFresh dense87.05 ± 0.2241.36 ± 0.31

Under the same REPRISE deployment policy, REPRISE training improves the scores by 11.03 and 2.32 points, with paired 95% confidence intervals of [3.03, 18.93] and [0.33, 4.35].

The dense student retains strong performance under fresh dense inference, but its score decreases when switched to sparse attention and cache reuse, particularly on RULER. This comparison supports training under the execution states used at deployment. Table 3 also reports every individual seed under both inference policies.

3. Initialization and optimization behavior — Appendix B.6

The teacher and dense student start from identical weights, and the adapter up-projections are initialized to zero. The first-step KL is recorded as zero for all three seeds, with gradient norms of approximately 2.5 × 10⁻⁷. Finite-precision gradients and AdamW updates move the students away from this initial point; the final KL values are 0.00152, 0.00144, and 0.00197. Appendix B.6 describes this identical-initialization dense self-distillation control and its transfer to the cached deployment policy.


Reviewer_NGmJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts a bidirectional diffusion language model to sparse attention and approximate KV-cache reuse. A student follows the deployed cache policy and is distilled from a dense teacher at the same saved token states. Selective replay reconstructs the initial prompt cache, a selected generation-refresh boundary, and its next active-block pass, enabling later prediction losses to train cache construction without retaining the full rollout graph. Inference combines less frequent prompt refreshes, more frequent generation refreshes, and sparse prompt attention. Experiments on UltraLLaDA-8B show clear RULER improvements over matched training controls and substantial inference speedups; gains on natural-language tasks are less conclusive.

Critical strengths
  1. The training contribution addresses a concrete mismatch between fresh-state training and inference with reused states. The replay construction is well specified, including its fixed discrete trajectory, layerwise reconstruction, and limited loss probes (Sections 5.2-5.3, pp. 5-6; Algorithm 2, p. 16). Because generation boundaries overwrite all generation KV entries, replay can bypass earlier active-block calls while preserving the dependencies of these probes. The reported exact-logit and gradient-intervention checks help substantiate this implementation.
  2. The RULER ablations are meaningful and carefully controlled. At 64K on 416 inputs, REPRISE improves over fresh sparse distillation by 1.38 points, with a paired 95% interval of [0.40, 2.43], and over detached prompt-cache training by 2.95 points, with [1.74, 4.23]; both gains hold for every matched training seed (Section 6.3, p. 9; Tables 8-10, pp. 18-19). This is stronger evidence for the training mechanism than comparisons with untrained cache baselines alone.
  3. The method uses only 64 rank-8 adaptation updates and merges them before deployment. Table 2 (p. 8) additionally tests complete generation under shared confidence-based parallel decoding, giving a 5.04x speedup over Dual on the 26-input RULER set at an 8.59-point higher score. The manuscript is unusually explicit about decoding, numerical precision, subset selection, uncertainty, and negative or inconclusive findings.
Critical weaknesses
  1. The evidence that cache-aware training improves natural-task quality is limited. On the 256-input LongBench subset, the gains over fresh sparse and detached-cache training are only 0.90 and 0.38 points, and both confidence intervals include zero (Tables 29-31, p. 32). All main V1 source-paired intervals also include zero (Table 21, p. 28). The V2 gains mostly arise on examples with unparseable baseline outputs, including failures of fresh dense NoCache, rather than demonstrating better reasoning (Tables 22-23, pp. 28-29). The paper acknowledges this, but it materially narrows what the headline average score gains establish.
  2. The scope of the headline speedups needs clearer qualification. The 48% TTFT and 80-82% TPOT reductions in the abstract correspond to the 61,088-token RULER probe in Table 26 (p. 30), measured once per method. This phase experiment fills only 129 of 256 positions (Appendix N.1, p. 31). Complete parallel-generation evidence is provided, but covers one checkpoint, 26 RULER inputs and 24 LongBench inputs at a single threshold; its gains differ substantially by workload (Tables 2 and 32, pp. 8 and 33). Thus "on average" in the abstract should not imply these particular percentages are benchmark-wide end-to-end averages.
  3. The system results establish a useful operating point more clearly than a quality-latency frontier. Table 3 (p. 9) varies the prompt refresh interval and sparsity, but reports only latency. The matched-inference training ablations isolate the value of the proposed training procedure, yet do not show whether a less aggressive, untrained refresh configuration could achieve comparable quality at similar cost. A compact score-and-latency comparison for the already considered H=32, 64, and 128 settings would make the practical trade-off easier to assess.
Questions for authors
  1. What explains the pervasive newline-only or otherwise unparseable V2 outputs, including for fresh dense NoCache? On the existing subset, can a decoding/prompt configuration verified against the backbone's native behavior clarify whether REPRISE retains an accuracy advantage once this failure mode is reduced? Please distinguish improved output validity from improved reasoning.
  2. Can the abstract explicitly tie its TTFT/TPOT percentages to the single RULER 64K phase probe and foreground the already reported complete-generation results? For the parallel experiment, how stable are the speedups to repeated timing on the existing inputs?
  3. Can task scores be added to the H=32/64/128 timing configurations, under a common precision/backend and matched inputs, to show the cost of obtaining the observed accuracy through more frequent refreshes versus adaptation?
7 comments
AuthorsOpenReview ↗

# Response to Reviewer NGmJ

Thank you for your careful assessments. We have updated the natural-language evaluations, repeated the complete-generation timings, and added quality and latency comparisons across refresh intervals.

W1 & Q1: Quality on natural-language tasks and the LongBench V2 evaluation

REPRISE aims to accelerate long-context inference while reducing the quality loss introduced by sparse attention and cache reuse, and improving quality where possible, rather than optimizing task quality independently of inference cost. Sections 1 and 6.2 explain this quality–efficiency objective. We have checked and updated the evaluations and tables as follows.

1. LongBench V2: Re-evaluation under a shared protocol — original Tables 22–23; revised Sections 6.1–6.2, Table 1, and Appendix L, Table 23

Under the previous long-form answer configuration, some outputs consisted of repeated newlines, including those from fresh dense NoCache. We therefore revisited the shared prompt and decoding settings. Keeping the original questions and context-truncation rules, we standardized the final answer prompt to Answer:, used FP16 computation and KV, zero temperature, a four-token output budget, and the same leading-option extraction rule for every method. We also checked the short-answer decoding path against the backbone’s native implementation on validation inputs. The protocol is specified in Section 6.1 and Appendix L.

We repeated the evaluation on the original 104 questions at all three input caps, including all three REPRISE training seeds. Every compared method now produces a valid option on every question. The accuracies in Table 1 are shown below; REPRISE reports the mean and sample standard deviation across Seed 1, Seed 2, and Seed 3.

Input capNoCachePrefixDualREPRISE
32K29.8132.6932.6934.62 ± 0.96
64K29.8135.5835.5835.90 ± 1.47
128K31.7335.5835.5834.94 ± 1.11

REPRISE’s mean accuracy exceeds NoCache at all three input caps and exceeds Prefix and Dual at 32K and 64K. At 128K, it is 0.64 percentage points below the two cache baselines. The V2 evaluation underlying original Tables 22–23 has been superseded by these shared-protocol results. The revised manuscript reports the overall comparison in Table 1 and all individual training seeds in Appendix L, Table 23.

2. LongBench V1: Mean scores and paired statistics — original Table 21; revised Section 6.2, Table 1, and Appendix L.1, Table 24

The main quality comparison in Section 6.2 uses the same 104 inputs for all methods within each setting. REPRISE has a higher mean score than Prefix and Dual at all three input caps and remains close to fresh dense NoCache:

Input capNoCachePrefixDualREPRISE
32K39.4937.7537.5638.32 ± 0.94
64K39.5237.6837.5138.38 ± 0.99
128K39.3136.8837.2838.90 ± 0.88

All nine source-paired score differences and their 95% confidence intervals from original Table 21 are retained in Appendix L.1, revised Table 24. Section 6.2 describes the observed mean performance and points to these paired statistics.

In the separate complete-generation comparison on 24 V1 inputs, Section 6.2 and Appendix P, Table 33 report a 1.55× speedup over Dual, with mean scores of 41.77 and 40.46, respectively. This comparison directly evaluates our objective of preserving or improving quality while accelerating inference.

3. Training ablations on natural-language tasks: Individual seeds, paired intervals, and task-level results — original Tables 29–31; revised Section 6.3, Figure 4, and Appendix O, Tables 30–32

For the training ablation on 256 LongBench inputs mentioned in your review, Section 6.3 and Appendix O report comparisons under the same deployment policy. All training methods are evaluated with three seeds on the same inputs. The following table summarizes the mean scores and paired differences reported in original Tables 29 and 30. Complete individual-seed results and paired intervals appear in revised Tables 30 and 31, respectively.

Training methodMean scoreREPRISE score gain over this methodPaired 95% confidence interval
Fresh sparse distillation39.4120+0.9008[−0.3604, 2.1836]
Detached prompt-cache training39.9326+0.3801[−1.3408, 2.0632]
REPRISE40.3128——

The eight-task breakdown from original Table 31 is retained in revised Table 32. Each task contains 32 inputs, and scores average the three training seeds, showing how performance varies across tasks.

Section 6.3 considers these results alongside the matched RULER training ablations when analyzing reused-state training and prompt-cache gradients. The natural-language results are reported through their measured means, task-level differences, and paired intervals.


W2 & Q2: Scope of the speedup claims and stability under repeated timing

Thank you for your suggestions on the scope of the efficiency claims and the stability of the timing results. We have replaced the single-input TTFT/TPOT percentages in the abstract with complete-generation results and repeated the measurements on the original inputs. The complete-generation evaluation covers RULER and LongBench V1/V2.

1. Complete-generation speedups — Abstract, Section 6.2, Table 2, Appendix P/Table 33, and Appendix P.2/Table 37

The revised abstract reports 5.05× and 1.55× complete-generation speedups over Dual on the RULER 32K and LongBench V1 evaluation subsets, respectively, under shared parallel decoding, together with an 8.59-point RULER score improvement.

The four settings with repeated timing against Dual are summarized below. Within each setting, both methods use the same inputs, hardware configuration, and decoding rules.

Evaluation settingInputsDual mean time (s)REPRISE mean time (s)Speedup
RULER 32K2643.548.625.05×
LongBench V1244.673.021.55×
LongBench V2, 64K cap10412.856.881.87×
LongBench V2, 128K capSame 104 questions32.7314.382.28×

Across these four settings, REPRISE achieves 1.55–5.05× complete-generation speedups, with a geometric mean of 2.40×. This aggregate weights the four settings equally, with LongBench V2 represented at two input caps.

These timing measurements are accompanied by task scores for the completed outputs. Section 6.2 and Appendix P, Table 33 report RULER scores of 74.87 for Dual and 83.46 for REPRISE, with both methods averaging 63.5 forward evaluations. Appendix P.2, Table 37 reports 35.58% accuracy for both methods at both V2 input caps, alongside REPRISE’s 1.87× and 2.28× speedups.

2. The original parallel-generation speedups remain stable under repeated timing — Sections 6.1–6.2 and Appendix P

To address your question directly, we repeated the measurements on the original 26 RULER inputs and 24 LongBench V1 inputs. For each input and configuration, an untimed exact-input warmup precedes three timed runs; the manuscript reports the average of the final two. Paired comparisons use the same physical GPU, BF16 arithmetic, and shared confidence-threshold and stopping rules.

Aggregating each repetition separately across inputs gives the following ratios of Dual’s mean time to REPRISE’s mean time:

Evaluation subsetFirst timed repetitionSecond timed repetitionThird timed repetition
RULER 32K5.082×5.051×5.056×
LongBench V11.566×1.545×1.549×

The speedups are consistent across the three repetitions, and the generated outputs are identical. The repeated measurements therefore support the original complete-generation speedup findings.

Appendix P, Table 34 (original Table 32) also reports paired 95% confidence intervals obtained by resampling inputs within tasks: [4.98, 5.12] for the RULER speedup over Dual and [1.29, 1.83] for LongBench V1. These intervals reflect variation across inputs, while the table above directly describes variation across timing repetitions.

The V2 64K/128K experiments use one untimed warmup followed by two timed repetitions, with latency averaged over the two measurements, as specified in Appendix P.2.

3. Scope of the original TTFT/TPOT measurements — Section 1, Appendix N.1, and Table 27 (original Table 26)

The original abstract’s approximately 48% TTFT and 80–82% TPOT reductions came from a single RULER input containing 61,088 tokens. We have replaced these headline percentages with the complete-generation results above and clarified the purpose and scope of the phase measurements in Section 1 and Appendix N.1.

Specifically, the phase experiment performs 128 fixed decoding steps after initialization, filling 129 of the 256 generation positions, to measure prefill and subsequent decoding costs. Its TTFT ends at the first token commitment at any generation position. Section 6.1 separately defines TTFT in the complete-generation experiment as the time to the first streamable output prefix.

Table 27 (original Table 26) retains the per-input phase measurements. Table 2 and Appendix P report complete-generation latency, task scores, and repeated-timing results, making the scope of each comparison explicit.


W3 & Q3: Quality–latency tradeoffs across refresh intervals

Thank you for suggesting that we examine refresh frequency, generation quality, and inference latency together. We have added task scores to the original timing table and included a complete-generation comparison between the pretrained model and REPRISE at H=32, 64, and 128, directly testing whether more frequent refreshes can match the quality and efficiency achieved through adaptation.

1. Comparing adaptation with more frequent refreshes under matched inference settings — Section 6.4 and Appendix A.6, Table 7

Using the original 26 RULER 32K inputs and 24 LongBench V1 inputs, we evaluate the pretrained model without REPRISE adaptation and REPRISE at three prompt-refresh intervals. H counts generation positions; a smaller H means more frequent prompt-cache refreshes.

All six configurations use the same sparse selection rule, BF16 weights and KV, BSR/SDPA executor, confidence threshold of 0.9, 32-token generation blocks, and stopping rule. Configurations are paired on the same physical GPU for each input. All REPRISE configurations use the same checkpoint trained at H=128, without retraining for evaluation at H=32 or H=64.

Each input and configuration receives one untimed warmup followed by three timed runs. We report the mean complete-generation latency of the final two runs:

ModelRefresh interval HRULER scoreRULER time (s)LongBench V1 scoreV1 time (s)
Pretrained model without adaptation3271.2228.4936.493.93
Pretrained model without adaptation6471.2215.2935.943.23
Pretrained model without adaptation12871.228.6835.022.87
REPRISE3285.9028.5742.643.96
REPRISE6483.4615.2441.973.25
REPRISE12883.468.6241.773.02

These results directly address whether increasing refresh frequency can replace adaptation:

  • RULER: At the same H=128, adaptation raises the score from 71.22 to 83.46 at similar latency. Reducing the unadapted model’s refresh interval to H=32 increases its latency to 28.49 seconds, while its score remains 71.22.
  • LongBench V1: More frequent refreshes improve the unadapted model’s score, but its H=32 configuration achieves 36.49 points in 3.93 seconds. REPRISE at the less frequent H=128 achieves 41.77 points in 3.02 seconds, with both a higher mean score and lower latency.

Thus, across the tested intervals and inputs, more frequent refreshes of the unadapted model do not recover REPRISE’s quality–latency operating point at H=128. Appendix A.6, Table 7 reports all six configurations, and Section 6.4 distinguishes this comparison from the original fixed-step experiment.

2. Adding task scores to the original timing configurations — Section 6.4, Table 4 (original Table 3), and Appendix A.5, Table 6

To address the missing quality measurements in the original table, we retained its inputs and fixed budget of 256 forward evaluations and added task scores for every configuration:

ConfigurationRefresh interval HRULER scoreMean time (s)
Dense attention with caching3271.5451.29
Dense attention with caching6471.5430.98
Sparse attention with caching, without adaptation6467.3723.51
REPRISE, Seed 112882.5617.15
REPRISE, Seed 212884.1017.19
REPRISE, Seed 312880.2617.06

Across the three seeds, REPRISE achieves a mean score of 82.31 and a mean latency of 17.13 seconds, giving a 2.99× speedup over dense caching at H=32.

This table reports quality and latency under the original fixed computation budget. The six-configuration experiment above further compares refresh frequency and adaptation under shared parallel decoding. Together, these results provide the requested assessment of the quality–efficiency tradeoff.

Comments on the paper

1 comment

General Response

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback. We appreciate the recognition of the training design, the prompt-cache gradient ablations, and the practical inference benefits. The reviewers' suggestions have helped us strengthen the experiments and clarify the contribution and scope of REPRISE. Below, we summarize the main revisions; our individual responses address each weakness and question in detail.

Our objective and contribution. REPRISE aims to accelerate long-context inference in globally bidirectional diffusion language models while reducing the quality loss introduced by sparse attention and cache reuse, and improving quality where possible. It trains the student under the deployed cache policy. Selective replay uses generation-refresh boundaries to reconstruct the computations needed by selected prediction losses, allowing these losses to update earlier cache construction without reconstructing the entire history. We have clarified this contribution and its relationship to prior work in Sections 2, 4, and 5.

The main experimental updates are as follows:

  1. Natural-language evaluation. We evaluated LongBench V2 with a validated direct-answer protocol shared by all methods, using the same 104 questions at 32K, 64K, and 128K input caps and three REPRISE training seeds. We updated the quality results and their interpretation, together with the LongBench training ablations and paired confidence intervals. These results appear in Sections 6.1–6.3, Table 1, and Appendices L and O.

  2. Related baselines and inference efficiency. We added Prefilling-dLLM and Focus-dLLM using their public implementations on RULER and LongBench V1/V2, reporting both quality and complete-request latency. We also repeated the complete-generation timing measurements: REPRISE achieves a 5.05× speedup on the 26-input RULER 32K subset and a 1.55× speedup on the 24-input LongBench V1 subset over Dual. The manuscript distinguishes these results from phase measurements and reports the quality–latency tradeoffs of the additional baselines. See Sections 6.1–6.2, Table 2, and Appendices C, N, and P.

  3. Selective replay correctness and training cost. We compared selective replay with activation-checkpointed full-history reconstruction using identical fixed trajectories and selected losses. All 42 paired gradient comparisons agree exactly in the completed configurations. For later-block probes with 512-token prompts, selective replay uses 8.7%–17.6% of the reference update time and 22.4%–46.6% of its peak GPU memory. We also report scaling with generation length and probe position. See Sections 5.3 and 6.3 and Appendix B.5.

  4. Training controls and refresh tradeoffs. We added a dense-student distillation control with matched training data, adaptation rank, and update count, evaluated under both fresh dense inference and the REPRISE deployment policy. We also added quality measurements to the H=32/64/128 refresh comparisons. See Sections 6.3–6.4, Tables 3–4, and Appendices A.6 and B.6.

Presentation and architectural scope. We reorganized the motivation, expanded the related-work discussion, and clarified the distinctions from Self Gradient Forcing, MAGE, and SparseD. We also discuss how sparse initial prefill could complement exact cache reuse in block diffusion, while identifying the attention architecture used in our experiments. See Sections 1–5 and Appendix K.

Revised manuscript and code. Additions and modifications are highlighted in blue in the revised manuscript, with margin notes connecting changes to reviewer comments. The anonymous REPRISE code repository is available at https://anonymous.4open.science/r/REPRISE-1454/.

We thank the reviewers and the Area Chair again for their careful assessment and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.