acceptodds
Under review as a conference paper at ICLR 2027

DStreamer: Acceptance-Aligned Diffusion Streaming for High-Throughput Serving and Online RLVR

Abstract

Long-form generation is increasingly the dominant cost in high-throughput LLM serving and online reinforcement learning with verifiable rewards (RLVR). Speculative decoding increases token progress per target-model invocation by verifying multiple proposals in parallel, but at throughput-oriented batch sizes its effectiveness depends on keeping proposal and state overhead low as concurrency and context grow. Online RLVR adds a distinct challenge: as the target policy changes during training, a drafter calibrated once can become progressively stale. We introduce DStreamer, a lossless speculative decoding framework with a single-pass diffusion-based in-model adapter that reuses the target model's cached KV state and requires no separate draft KV cache. DStreamer warm-starts the adapter with consistency distillation, then optimizes proposal paths with an acceptance-aligned reinforcement-learning objective under autoregressive verification, directly targeting the prefix-sensitive criterion that determines speculative utility. We evaluate DStreamer in the throughput-oriented regimes where speculative decoding is most constrained by proposal and state overhead: continuous-batched serving, long-form generation, and synchronous online RLVR. Across Qwen and GPT-OSS models, DStreamer expands the serving frontier under concurrencies 64-1024, outperforming DFlash and EAGLE-3 throughout the evaluated range and reaching up to higher aggregate throughput than autoregressive decoding at matched per-request rate, while requiring no separate draft KV cache. DStreamer++ extends acceptance-oriented training to evolving policies while preserving exact on-policy rollouts, achieving - end-to-end RLVR speedup across Qwen3-4B, 8B, and 14B under synchronous H100 training, while preserving final policy quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, diffusion language models, speculative streaming, reinforcement learning with verifiable rewards, high-throughput LLM serving, on-policy reinforcement learning, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ms8dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DStreamer and DStreamer++, a speculative decoding framework for high-concurrency LLM serving and online RLVR. Building on the shared-KV adapter design of Speculative Streaming (which drafts tokens in parallel without a separate draft KV cache), DStreamer warm-starts the adapter with masked diffusion consistency distillation and then fine-tunes it via GRPO on complete candidate paths to maximize autoregressive acceptance. To handle policy shifts during online RLVR, DStreamer++ continuously updates the adapter on rollout windows. Experiments on Qwen3 and GPT-OSS show up to 3.8×3.8\times serving throughput gains over AR (outperforming EAGLE-3 and DFlash) and 1.93–2.15×1.93\text{--}2.15\times end-to-end RLVR speedup with no loss in final accuracy.

Critical strengths
  1. Practical algorithm-system co-design for online RLVR and high-batch decoding. Reusing the target KV cache directly avoids the memory bottleneck of separate draft models at large batch sizes and long contexts. More importantly, DStreamer++ makes online drafter RL practical in synchronous FSDP training: by sampling multiple candidate paths from a single adapter forward pass and reusing the rollout KV cache in place, online adaptation adds only ~3.8% step-time overhead while effectively preventing acceptance drop over training.

  2. Strong empirical validation in realistic high-throughput regimes. Rather than focusing on batch-size-1 latency, the paper evaluates continuous batching up to concurrency 1024, context scaling up to 8K (where baseline drafters OOM), and multi-seed end-to-end RLVR runs measured in wall-clock GPU-hours vs. benchmark accuracy. This convincingly shows that the proposed method delivers real wall-clock speedups where rollout cost actually dominates.

Critical weaknesses
  1. Training gains are not cleanly separated from the inherited shared-KV architecture. DStreamer adopts its in-model adapter and target-KV reuse directly from Speculative Streaming, so the core novelty lies in the two-stage training recipe. However, the main serving and RLVR benchmarks do not include either the original Speculative Streaming training method or the Stage-1 consistency-distilled checkpoint as baselines. While Figure 3 shows that RL improves over the distilled initialization on a validation set, it does not show how much this translates into actual wall-clock throughput in the main benchmarks. We need a controlled comparison across the original Speculative Streaming recipe, the Stage-1 distilled adapter, and the final RL-trained adapter on the same architecture and data (with matched training budgets), reporting both accepted length τ\tau and end-to-end throughput. Without this, it is hard to tell how much of the advantage over EAGLE-3 and DFlash comes from the proposed training versus the zero-KV architecture itself.
Questions for authors
  1. Controlled ablation of the two-stage training recipe on the same shared-KV architecture: Since DStreamer uses the same inference architecture and KV-sharing mechanism as Speculative Streaming, could you provide a controlled comparison across (i) the original Speculative Streaming training recipe, (ii) the Stage-1 consistency-distilled checkpoint (before RL), and (iii) the full Stage-2 RL-trained DStreamer adapter on the same architecture and training data (with training compute/steps matched or clearly reported)? Specifically, how much does each stage improve mean accepted length (τ\tau) and actual vLLM serving throughput on the main benchmarks (e.g., Qwen3-4B on GSM8K and HumanEval)? Seeing these numbers would directly clarify how much of the speedup over EAGLE-3 and DFlash comes from your training recipe versus the zero-KV adapter design itself.
2 comments

We appreciate the reviewer's suggestion to isolate the training contributions from the inherited shared-KV architecture. We have added a dedicated Component Analysis (Sec. 4.2, Fig. 3; Appendix E.1, Fig. 12), reporting both accepted length and vLLM serving throughput on GSM8K and HumanEval. We compare Speculative Streaming-style direct CE training, consistency-distilled diffusion, and their RL-optimized counterparts using the same shared-KV adapter architecture and training data. Both initializations use 30K optimizer steps, followed by the same 10K-step acceptance-aligned RL procedure.

The controlled comparisons show how each training stage contributes:

  1. Diffusion improves initialization. Training on partially masked windows provides an easier reconstruction task than predicting all future tokens at once. Consistency distillation transfers this benefit to single-pass drafting, improving accepted length from 3.08 to 3.33 on GSM8K (Fig. 3a) and 2.58 to 2.69 on HumanEval (Fig. 12b).

  2. Acceptance RL builds on this advantage. By optimizing complete proposal paths for prefix compatibility rather than individual token predictions, RL further improves both initializations. Diffusion retains its advantage after RL: 3.74 vs. 3.59 on GSM8K (Fig. 3c) and 2.83 vs. 2.72 on HumanEval (Fig. 12b).

  3. Both stages improve serving throughput. Full DStreamer achieves the highest vLLM throughput across evaluated batch sizes on GSM8K (Fig. 3b) and HumanEval (Fig. 12a). At batch size 256 on HumanEval, throughput increases from approximately 13.1K tokens/s with direct CE to 13.9K after diffusion initialization, and 14.6K after acceptance RL. In comparison, direct CE + RL reaches 14.01K tokens/s.

By holding the shared-KV architecture fixed, these comparisons isolate the acceptance and serving-throughput improvements attributable to each training stage. Diffusion provides a stronger single-pass initialization, while acceptance-aligned RL further improves its usefulness under autoregressive verification.

Thank you for the additional experiments and clarification, the ablations address my concern about the attribution of the two-stage training gains. After considering the updated evidence and overall strengths, limitation of the paper, I will maintain my original score.

Reviewer_p752OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes DStreamer, a speculative decoding framework that uses a single-pass diffusion-based adapter with shared target model KV cache. The adapter is initialized through masked diffusion and consistency distillation, then optimized using an acceptance-oriented RL objective. The authors further introduce DStreamer++, which continuously updates the adapter during online RLVR to address speculative drift caused by changes in the target policy. The paper reports improvements in serving throughput and up to 2.15× end-to-end RLVR speedup while maintaining comparable final model performance.

Critical strengths
  1. The paper address an important practical problem: accelerating rollout generation during online RLVR.
  2. The experiments cover multiple model sizes and report both inference throughput and end-to-end RLVR efficiency.
Critical weaknesses

1. Insufficient comparison between acceptance-oriented RL and conventional distillation objectives.

One of the central contributions is optimizing the drafter with an acceptance-oriented RL objective instead of conventional token-level supervision. However, the paper does not provide a direct comparison against CE/KL-based distillation under the same architecture and training budget.

In particular, the proposed soft reward is based on the target model's log-probabilities, which is closely related to likelihood-based distillation. It is therefore unclear how much of the improvement comes specifically from the acceptance-oriented objective rather than simply adapting the drafter to the target distribution.

A controlled comparison between online CE/KL distillation and acceptance-oriented RL would substantially strengthen the paper's central claim.

2. The necessity of diffusion-based initialization is unclear.

Although the framework is presented as diffusion-based, the adapter generates speculative tokens in a single forward pass at inference, following consistency distillation.

This makes the resulting mechanism appear similar to parallel multi-token prediction. It is unclear why diffusion-based training is necessary instead of directly training a parallel prediction head using conventional supervised objectives.

An ablation comparing direct parallel token prediction, diffusion-based initialization, and their respective performance after acceptance-oriented RL would help clarify the contribution of diffusion.

3. Limited controlled comparisons with online-adaptive speculative decoding methods.

The paper compares against DFlash, EAGLE-3, and other speculative decoding baselines. However, because an important contribution of DStreamer++ is its ability to adapt to an evolving target policy, comparisons against frozen drafters alone are insufficient.

Although the authors include results for online EAGLE-3 and EfficientRollout, these are taken from their respective papers rather than reproduced under the same experimental setting. Related approaches such as ReSpec and FastGRPO, which also address drafter adaptation during RL training, are discussed but not directly evaluated.

Controlled comparisons with these methods would provide stronger evidence for the effectiveness of the proposed online adaptation strategy.

4. The presentation makes the core contributions difficult to identify.

Overall, I found the paper somewhat difficult to follow. Architectural details, diffusion-based initialization, acceptance-oriented optimization, and online RLVR adaptation are frequently interleaved.

In particular, it is not always clear which components are inherited from Speculative Streaming and which constitute the novel contributions of this paper. The role of diffusion is also difficult to understand given the single-pass inference mechanism.

A clearer decomposition of the framework and a more explicit distinction between existing components and proposed improvements would significantly improve readability.

Questions for authors

see weakness

4 comments

We have reorganized Sec. 3 to clearly distinguish the inherited speculative decoding mechanism from our training contributions, and added detailed ablations of initialization, acceptance optimization, and online adaptation (Sec. 4.2, Figs. 3–4; Sec. 4.4, Figs. 6–7). Our central contribution is how we train parallel speculation for autoregressive verification. Building on Speculative Streaming's single-pass, shared-KV drafting mechanism, we introduce three contributions:

  1. Diffusion-based initialization: Learns from partially observed token windows, with consistency distillation transferring this benefit to single-pass drafting.
  2. Acceptance-aligned optimization: Trains complete proposal paths for consecutive-prefix acceptance, which conventional token-wise supervision does not directly optimize.
  3. Online acceptance-aligned adaptation (DStreamer++): Extends this objective to evolving target policies during RLVR. Controlled comparisons with online CE/KL updates and reproduced adaptive baselines demonstrate its advantage over conventional online drafter training approaches.

1. Acceptance-oriented RL versus conventional distillation

We have added controlled comparisons against continued CE and forward-KL target-logit distillation, starting from the same consistency-distilled checkpoint with identical adapter architecture and 10K training steps (Fig. 4a).

While the reviewer correctly notes that our soft reward uses target log-probabilities, the learning process differs from conventional distillation. KL trains the drafter to match token-wise target distributions. DStreamer instead samples complete paths from its current drafter, evaluates them under the target's autoregressive verifier, and uses GRPO to reinforce higher-reward paths. This lets the drafter learn from the acceptance of its own proposals and favor token combinations that yield longer accepted prefixes, rather than only matching target distributions position-wise.

Importantly, even without the target-likelihood soft reward, hard-only RL reaches τ = 3.70, versus 3.45 for KL and 3.56 for soft-only RL. Combining both rewards reaches 3.74 (Fig. 4a). On GSM8K, RL improves first-token survival by just 2 percentage points over KL, but three-token prefix survival by 19 points (Fig. 4b). These results show that the benefit lies in learning more usable proposal paths, not merely improving token-wise distribution matching.

2. Why diffusion-based initialization?

We compare direct parallel CE and diffusion-based initialization under the same shared-KV architecture, with 30K initialization steps and the same subsequent 10K-step RL procedure (Fig. 3).

Direct CE predicts all future tokens from fully masked positions. Diffusion instead learns through easier conditional reconstruction tasks, using partially visible tokens to predict the remaining ones across different masking levels. Iterative three-pass diffusion achieves higher acceptance (3.64 vs. 3.08 for CE), but requires multiple drafter inference passes. Consistency distillation transfers much of this benefit into a single-pass drafter (3.33), improving the acceptance–compute tradeoff (Fig. 3a,b).

The initialization advantage also persists after acceptance RL (3.74 vs. 3.59; Fig. 3c) and translates into higher vLLM throughput (Fig. 3b). Thus, diffusion improves proposal quality during training, while distillation preserves efficient single-pass inference.

3. Controlled comparisons with online-adaptive speculative decoding

We have expanded the online RLVR evaluation with two sets of comparisons (Figs. 6–7).

First, we compare DStreamer++ against online CE and target-logit KL using the same adapter, initialization, retained rollout windows, minibatch configuration, and update schedule. While both improve over frozen drafting, acceptance-aligned RL achieves higher end-to-end speedup across Qwen3-4B, 8B, and 14B (Fig. 7a). As the target policy evolves, DStreamer++ continually realigns its proposals using acceptance feedback, helping sustain accepted-prefix length (Fig. 11) and rollout throughput throughout RLVR (Fig. 6).

We also reproduced ReSpec, FastGRPO, and online EAGLE-3 under the same Qwen3-14B/8K/8×H100 RLVR setup, keeping the target-policy training configuration, prompt batch size, rollout group size, sampling settings, and retained-window budget fixed (Fig. 7b).We have now reproduced all three baselines in our experimental setup, replacing the earlier cross-paper comparisons. DStreamer++ achieves 2.15× end-to-end speedup, outperforming ReSpec (2.01×), the strongest reproduced external adaptive baseline.

4. Clarifying the presentation

We reorganized Sec. 3 to distinguish the inherited shared-KV architecture from our contributions: diffusion initialization (3.2), acceptance-aligned optimization (3.3), and online adaptation (3.4). We clarify why diffusion benefits training despite single-pass inference, while Figs. 3–4 isolate the gains from each stage.

We have also completed an additional controlled analysis of acceptance-aligned optimization during online RLVR and updated the manuscript with the new results (Appendix Fig. 13).

To complement our earlier fixed-target ablations, we measure verification-prefix survival throughout online RLVR for DStreamer++, online CE, and online KL under matched adaptation settings on Qwen3-14B/8K. The results show a clearer advantage for DStreamer++ when considering longer accepted prefixes. At step 250, DStreamer++ achieves 23.8% three-token prefix survival, compared with 16.6% for online KL and 15.9% for online CE. At step 500, although first-token survival is comparable between DStreamer++ and KL (56.3% vs. 55.2%), three-token prefix survival remains higher for DStreamer++ (22.8% vs. 16.9%). These results provide further evidence that acceptance-aligned optimization improves longer-prefix survival beyond conventional online CE/KL adaptation, even when first-token acceptance is comparable.

We would appreciate your assessment of whether this additional evidence, together with the revised experiments, addresses your central concerns. We would be glad to clarify any remaining methodological or empirical questions.

Reviewer_p752OpenReview ↗

Dear authors

Thank you for the detailed response and additional experiments. The new results address most of my empirical concerns and strengthen the paper. However, I still have reservations about the methodological novelty relative to Speculative Streaming, as the main contributions appear to focus on training improvements rather than a fundamentally new approach. I will therefore raise my rating from 1 to 2.

Thank you for considering our response and the additional experimental evidence. We would like to further clarify the methodological contributions relative to Speculative Streaming, particularly why optimizing how speculative proposals are learned is as important as how they are generated. We have also added new cross-architecture experiments (Fig. 15) demonstrating that our acceptance-aligned optimization extends beyond Speculative Streaming.

1. Learning as a core component of speculative decoding. Speculative decoding efficiency depends on two factors: the cost of generating proposals and how many tokens survive each verification step. Speculative Streaming primarily addresses the former through efficient single-pass drafting, whereas DStreamer addresses the latter through a different learning methodology. Our two-stage approach uses diffusion-based initialization to learn stronger parallel proposals, followed by acceptance-aligned optimization to improve prefix survival under autoregressive verification. Retaining the shared-KV architecture is deliberate: we improve acceptance without introducing additional inference-time drafting passes or model state (Figs. 3–4).

2. Generalizability across drafting architectures. Our acceptance-aligned optimization is not specific to Speculative Streaming. To demonstrate this, we added experiments applying the method to DFlash and EAGLE-3 while preserving their respective drafting mechanisms. On GSM8K, accepted length improves from 3.26 to 3.58 for DFlash and from 2.48 to 3.05 for EAGLE-3 (new Fig. 15). Importantly, these gains extend from parallel block drafting (DFlash) to autoregressive drafting (EAGLE-3), showing that the proposed learning principle is applicable beyond the inherited single-pass architecture.

3. Acceptance-aligned adaptation during online RLVR. While online drafter adaptation has been explored before, DStreamer++ extends path-level acceptance optimization to an evolving target policy. Unlike conventional online CE/KL adaptation, which focuses on matching the changing target distribution, our approach directly optimizes accepted-prefix quality as the target evolves. Matched comparisons show substantially better survival of longer prefixes, even when first-token acceptance is comparable (Appendix Fig. 13). These improvements also translate into higher measured end-to-end RLVR speedup than the reproduced online-adaptive baselines (Fig. 7).

Together, these results show that DStreamer's contribution is not limited to improving Speculative Streaming's training. Our approach addresses a fundamental aspect of speculative decoding, how to learn proposals that maximize progress under autoregressive verification with demonstrated benefits across different drafting architectures and evolving target policies. We hope this addresses your concern about methodological novelty and would appreciate consideration of the additional evidence.

Reviewer_4fV9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DStreamer, a lossless speculative decoding method for throughput-oriented LLM serving, and DStreamer++, which extends the approach to online reinforcement learning with verifiable rewards (RLVR). DStreamer employs a single-pass, in-model speculative adapter that reuses the target model's KV cache instead of maintaining a separate draft KV cache. It combines masked-diffusion-based initialization with a path-level, acceptance-aligned reinforcement learning objective, addressing the mismatch between parallel token proposals and autoregressive, prefix-sensitive verification. DStreamer++ updates this adapter as the target policy evolves during RLVR while leaving the distribution of generated rollouts unchanged under lossless verification.

The experiments cover Qwen3 and GPT-OSS models, continuous-batched serving, long-form generation, and synchronous RLVR. The paper reports up to 3.8x aggregate serving throughput at a matched per-request generation rate, along with 1.93–2.15x end-to-end RLVR speedups and similar final task accuracy.

Critical strengths
  1. The paper identifies an important mismatch between factorized parallel proposals and autoregressive, prefix-sensitive acceptance, and proposes a coherent way to optimize the quantity that matters at verification time. Combining acceptance-aligned path training with an in-model adapter that reuses the target KV state is a practical design for high-concurrency and long-generation workloads. The losslessness argument further helps establish that speculative acceleration need not alter the target sampling distribution.

  2. The evaluation is unusually relevant to real serving and RL training. Rather than relying only on average acceptance length or fixed-batch microbenchmarks, the paper reports throughput-versus-per-request-rate frontiers under continuous batching (Figure 4), end-to-end RLVR training speedups (Figure 6), and accuracy-versus-compute comparisons (Figure 7). The speculative-drift analysis and measured adapter-update overhead are also useful. The results provide credible evidence of meaningful systems-level gains across the evaluated configurations.

Critical weaknesses
  1. The contribution of the new training objective would benefit from a more direct, controlled comparison against the original Speculative Streaming training approach using the same adapter architecture, data, and training budget. Figure 3 and the reward ablations support the value of acceptance-oriented optimization, but a matched end-to-end serving comparison would make it easier to distinguish improvements from the new training procedure from those attributable to the inherited shared-KV architecture.

  2. In online RLVR, the current evidence clearly shows that adapting the drafter is beneficial relative to freezing it, but the specific advantage of acceptance-aligned adaptation over a similarly budgeted online distillation or supervised update is less clear. Moreover, the authors note that some published online baselines are not rerun under identical conditions (Appendix D.3). These limitations primarily affect the attribution and breadth of the comparative claims rather than undermining the main demonstrated speedups.

Questions for authors
  1. Could the authors include a matched-architecture and matched-training-budget comparison with the original Speculative Streaming training approach, preferably reporting both accepted length and end-to-end serving throughput? A concise component-level ablation separating consistency distillation from acceptance-aligned RL would also clarify where the improvements arise.

  2. Can the authors compare DStreamer++ against online supervised or distillation-based updates to the same adapter with a matched adaptation budget? This would help establish whether the gain comes primarily from continuous adaptation or from the particular acceptance-aligned objective. Please also clearly distinguish same-system measurements from cross-paper baseline results in the end-to-end RLVR comparison.

2 comments

We appreciate the reviewer's emphasis on isolating the training contributions. We have added a dedicated Component Analysis (Sec. 4.2, Figs. 3–4) and controlled online-adaptation comparisons (Figs. 6–7) to address both concerns.

1. Separating training improvements from the inherited shared-KV architecture

Our component analysis compares Speculative Streaming-style direct parallel CE training, diffusion/consistency initialization, and their RL-optimized counterparts under the same shared-KV adapter architecture. Both initialization strategies use 30K optimizer steps, followed by the same 10K-step acceptance-aligned RL procedure.

  • Initialization: Partially masked reconstruction provides an easier learning task than direct parallel prediction. Consistency distillation transfers this benefit to single-pass drafting, improving accepted length from 3.08 to 3.33 (Fig. 3a).
  • Acceptance optimization: RL directly optimizes usable prefixes rather than isolated token predictions, improving accepted length from 3.33 to 3.74 after diffusion initialization, compared with 3.59 for direct CE + RL (Fig. 3c).
  • Serving throughput: These improvements translate into higher vLLM throughput, with full DStreamer performing best across evaluated batch sizes under the same inference architecture (Fig. 3b).

Since the inference architecture and KV-reuse mechanism are held fixed, these comparisons isolate the benefits of our training procedure from those of the inherited design.

2. Isolating the benefit of acceptance-aligned online adaptation

We have added online CE and target-logit KL baselines using the same adapter, initialization, retained rollout windows, minibatch configuration, and update schedule as DStreamer++ (Figs. 6–7a). Both improve over frozen drafting, but DStreamer++ achieves higher end-to-end speedup across all three Qwen3 model sizes.

The key difference is how the drafter learns. CE/KL matches token-wise targets, whereas DStreamer++ samples complete paths from its current drafter, evaluates them under the autoregressive verifier, and uses GRPO to reinforce higher-reward paths. This allows the drafter to learn from its own proposals and favor token combinations that survive consecutive verification, rather than simply imitating the target at each position.The fixed-target ablation (Fig. 4) supports this distinction: relative to KL, acceptance-aligned RL improves GSM8K first-token survival by just 2 percentage points, but three-token prefix survival by 19 points.

We have now reproduced ReSpec, FastGRPO, and online EAGLE-3 under the same Qwen3-14B/8K/8×H100 RLVR setup, replacing the earlier cross-paper comparisons (Fig. 7b). Including drafter-update overhead, DStreamer++ achieves 2.15× end-to-end speedup versus 2.01× for ReSpec, the strongest reproduced external adaptive baseline.

Thank you for the clarifications and additional experiments. The reported component ablations and online CE/KL comparisons substantially address my concerns about attribution, and the reproduced baselines strengthen the evaluation. I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.