acceptodds
Under review as a conference paper at ICLR 2027

Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion–AR Self-Speculation Frontier

35 views

Abstract

Diffusion drafting with autoregressive (AR) verification has emerged as a promising paradigm for efficient speculative decoding. Recent self-speculation models, represented by Nemotron-Labs-Diffusion, further simplify the speculative pipeline by unifying drafting and verification within a shared backbone, while enabling longer acceptance lengths. However, the Pareto frontier between aggregate and per-request throughput remains underexplored. At low concurrency, sequential draft–verify execution requires two model forward passes per round, limiting the effective tokens per forward (TPF). By contrast, at high concurrency, longer drafts incur increasingly expensive computation, forcing individual requests to operate under constrained speculation budgets and preventing full exploitation of the full-backbone drafter. Our key observation indicates that drafting and verification exhibit reciprocal predictability. Draft logits can anticipate likely verification mismatches, while recent verification outcomes predict future drafting utility and suitable block sizes. Building on this observation, we introduce **Reciprocal Guidance** (**RecGuide**), a runtime draft–verify orchestration framework that adapts speculative decoding to varying serving loads. RecGuide exploits spare compute capacity through verification-overlapped drafting at low concurrency, while dynamically allocating request-specific draft block sizes as the workload becomes increasingly compute-intensive. Experiments across a wide range of concurrency levels demonstrate consistent throughput improvements over vanilla self-speculation, achieving up to 1.8× speedup.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Diffusion Language Models, Self-Speculation, LLM Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hn6COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RecGuide, a runtime orchestration framework for shared-backbone diffusion–AR self-speculation. At low concurrency, draft distributions predict likely verification failures and correction tokens, enabling prospective next-round drafts to be co-batched with current verification. At higher concurrency, historical verifier entropy, previous block size, and acceptance outcomes guide block-size selection, while bucketed execution reduces padding.

Critical strengths

In this paper, predictive correction and continuation branches allow next-round drafting to be co-batched with current verification, addressing the bottleneck of two sequential full-model forwards. End-to-end throughput measurements demonstrate practical benefits and reveal the importance of execution strategy.

Critical weaknesses
  1. The low- and high-concurrency mechanisms are evaluated separately at fixed loads, without demonstrating unified switching under dynamic loads. Moreover, a substantial portion of the Table 6 gain comes from bucketing, while simple adaptive policies using the same backend are absent. It is therefore difficult to separate the contributions of reciprocal prediction, execution optimization, and scheduling configuration to the claimed serving-frontier improvement.
  2. The lightweight-drafter comparison is not matched. DFlash is trained only on ShareGPT with the SpecForge recipe because of budget limits, whereas LinearSS, TiDAR, and RecGuide use the official NLD backbone.
Questions for authors
  1. With identical KV handling, kernels, and bucketing, how much improvement remains over a simple block expansion/contraction policy based on the previous acceptance length?
  2. After training DFlash to convergence on data comparable to the NLD drafter, does its single-request speedup remain clearly below RecGuide?
  3. Does throughput rise or fall when predictive overlap and adaptive block sizing are enabled together?
3 comments
AuthorsOpenReview ↗

Response to Reviewer hn6C: Part 1

Dear Reviewer hn6C,

We sincerely thank you for highlighting the practical benefits of co-batched drafting and verification. We also appreciate your suggestions regarding the routing comparison, DFlash reproduction, and joint execution. Below are our responses to your concerns.

1. Fair Comparison with Simple Block-switching Strategies

We agree that comparing with simpler block-size policies under identical KV management, kernels, and bucketing is important. On NLD-8B, we considered three additional baselines. (a) Direct chooses the block-size bin from the previous round's acceptance length. (b) Direct + Full Signal uses the same direct mapping and adds the previous round's full-acceptance indicator. (c) Calibrated + Full Signal uses the same calibration set as RecGuide to calibrate the mapping from the previous acceptance length to a block-size bin and also uses the full-acceptance indicator. The table reports the average throughput change relative to fixed-block serving across the tested concurrency levels.

PolicyAverage throughput change
Direct-21.80%
Direct + Full Signal-0.23%
Calibrated + Full Signal+34.82%
RecGuide+39.84%

The two uncalibrated policies have difficulty adapting to fluctuations in acceptance length during decoding. They yield little or negative throughput gain. The full-acceptance signal helps the router expand short blocks when the whole draft is accepted. Using the calibration set to map acceptance length to the next block size further improves throughput. RecGuide still leads the strongest calibrated baseline by 5.02 percentage points with the same backend.

We emphasize that the large gain after adding bucketing in Table 6 is a joint gain from adaptive drafting and bucketing. Adaptive drafting saves computation by choosing request-specific draft lengths. Within a mixed-length batch, shorter drafts would be padded to the longest block, eroding those savings. Bucketing groups requests by block size and preserves the savings from the routing policy, making it essential for adaptive drafting to deliver end-to-end throughput gains.

2. Training Data and Convergence of the Reproduced DFlash

We compared the sample and token counts of the training data that Z-Lab released for DFlash, the ShareGPT data used in our reproduction, and the training data described in the NLD report. For DFlash, we count tokens over six epochs. Our training configuration follows the official recipe of SpecForge in the SGLang community, and the amount of training data is on the same scale as Z-Lab's release.

Training dataZ-LabShareGPTNLD
Samples (K)100121—
Tokens (B)0.5980.531300

Matching NLD's training data presents two challenges. First, the scale is large. The NLD report describes training on 256 NVIDIA H100 GPUs. Second, those 300B tokens are used for joint AR and diffusion training of the shared backbone. NLD uses that backbone for drafting as well as verification. DFlash trains a lightweight drafter separately from the target model, at the smaller scale needed for a dedicated drafter.

To check how much further the reproduced drafter could improve with training, we measured acceptance length after each epoch on NLD-8B.

Epoch123456
τ2.763.553.974.154.304.40

The results show a clear convergence trend. RecGuide delivers approximately 1.5× the single-request end-to-end throughput of our DFlash reproduction. Given the convergence trend and the size of this gap, we expect that additional epochs or a larger dataset such as Open-PerfectBlend would still leave DFlash well below RecGuide.

AuthorsOpenReview ↗

Response to Reviewer hn6C: Part 2

3. Strategy Switching and Joint Execution

Following DFlash and NLD, we reported single-request and batched results separately in the original submission. Because RecGuide uses different serving strategies at different loads, this presentation gave the impression that Predictive Draft-Verify Overlap (PDO) applies only at C = 1 and that the system must switch to Verification-Guided Adaptive Drafting (VGD) at C = 2.

We compared PDO alone, VGD alone, and the two strategies together at the same concurrency levels on NLD-8B.

Aggregate tok/sC = 248163264128
PDO1781303946606631719574808108
VGD159528274835741892271025810093
Joint1796293645696291739582167230

Joint execution offers little benefit over PDO at low load. It is slightly ahead of PDO at C = 2 and falls behind from C = 4 through C = 16. In this range, VGD's relatively conservative block-size choices give up potential acceptance opportunities that PDO could exploit. At C = 32 and 64, VGD offsets some of PDO's additional computation. VGD alone still leads the combination. At the highest concurrency, C = 128, the combination is slower than either strategy alone. This shows how strongly the two strategies can interfere when both draw on the same compute budget, given that each was designed for a different load regime.

These results place the switching boundary between C = 4 and 8 for the GPU and workload used in our experiments. GPU compute-to-bandwidth ratios vary substantially, and acceptance length depends on the task. RecGuide therefore adapts the switching boundary to the hardware and workload, analogous to the task- and hardware-specific adaptation in DSpark's Hardware-Aware Prefix Scheduler.

Thank you for your questions about throughput gains, DFlash training, and joint execution, which helped us strengthen the comparisons and clarify the role of each strategy. We hope the responses above can address your concerns and contribute to a reconsideration of review score. Looking forward to discussing more with you.

Best,
Authors

AuthorsOpenReview ↗

Comment to Reviewer hn6C

Dear Reviewer hn6C,

Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above incorporating all requested experiments, analyses, and clarifications.

Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!

Reviewer_mopYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work builds on self-speculative decoding, in which a single model serves as both drafter and verifier. The paper proposes coupling the two stages of inference to improve throughput in both low- and high-concurrency settings. At low concurrency, the system uses the draft’s uncertainty (the gap between its top two token candidates) to prepare likely follow-on drafts (i.e., a correction branch and a continuation branch) and runs them in the same batch as verification of the current draft. If verification confirms a prediction, the next draft is already available, and a serial forward pass is skipped. At high concurrency, the system uses feedback from verification (recent verifier entropy and whether the last block was fully accepted), together with a profiled cost model of the current load, to choose a per-request draft block size. Requests are then grouped into buckets by block size so batches stay uniform. The paper evaluates in the context of NLD-8B and 14B and shows gains at both low-concurrency and high-concurrency settings.

Critical strengths

Paper builds on empirical observations (e.g., impact of top-2 margin and acceptance length persistence) to build out a system that is then fleshed out with some necessary and useful systems techniques (e.g., shared-prefix KV access, padding-free execution, block-size bucketing) to help it demonstrate performance gains.

Critical weaknesses

The correction branch needs to be reasonably correct in order to get the value of pairing up additional draft steps with the ongoing verification step. This makes the paper's choice of using greedy decoding for its evaluations, as opposed to non-deterministic temperature sampling, somewhat questionable and could end up overestimating the gains from this approach.

Questions for authors
  • How would the system perform if temperature sampling is used instead of greedy decoding?
  • The transition from low concurrency to high concurrency seems to be between 1 and 2. Can you justify that very distinct transition?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mopY

Dear Reviewer mopY,

We sincerely thank you for the positive evaluation and your thoughtful questions regarding temperature sampling and strategy switching. Below are our responses to your concerns.

1. Efficiency with Temperature Sampling

We followed the original LinearSS evaluation and used greedy decoding in the manuscript, reflecting the setting for which the Nemotron-Labs-Diffusion models were primarily trained and released. We have now also evaluated T = 1.0, a configuration widely used in speculative decoding studies. At this temperature, RecGuide achieves average speedups of 1.80× on NLD-8B and 1.85× on NLD-14B, ahead of 1.46× and 1.56× for LinearSS. RecGuide increases τ and achieves a higher speedup in all 16 model-benchmark pairs.

ModelMethodMetricGSM8KMATH-500AIME25HumanEvalMBPPLCBGPQAIFEvalAvg.
NLD-8BLinearSSτ6.196.235.785.785.454.723.933.255.17
NLD-8BLinearSSSpeedup2.05×1.69×1.32×1.83×1.81×1.09×0.89×1.04×1.46×
NLD-8BRecGuideτ6.827.516.666.766.324.974.293.345.83
NLD-8BRecGuideSpeedup2.13×2.31×1.95×2.14×2.00×1.49×1.31×1.09×1.80×
NLD-14BLinearSSτ6.115.795.096.225.334.544.203.265.07
NLD-14BLinearSSSpeedup2.12×1.72×1.32×2.17×1.89×1.16×1.03×1.10×1.56×
NLD-14BRecGuideτ6.716.565.276.875.474.934.413.425.45
NLD-14BRecGuideSpeedup2.29×2.20×1.75×2.35×1.91×1.63×1.44×1.21×1.85×

We analyzed why the gains over LinearSS are slightly smaller than with greedy decoding. The focus on greedy decoding in NLD's training and release helps explain why the models themselves have lower acceptance under stochastic sampling. Verifier mismatches occur earlier and more often. This makes the first rejected position harder to predict and leaves fewer opportunities to reuse the limited set of drafts prepared in parallel. The gains across model sizes and benchmarks show that the correction branch remains effective under stochastic sampling. We expect this strategy to deliver larger gains with future self-speculation models that are trained more extensively for this setting.

2. Strategy Switching Across Concurrency Levels

Following DFlash and NLD, we reported single-request and batched results separately in the original submission. Because RecGuide uses different serving strategies at different loads, this presentation gave the impression that Predictive Draft-Verify Overlap (PDO) applies only at C = 1 and that the system must switch to Verification-Guided Adaptive Drafting (VGD) at C = 2.

We evaluated PDO and VGD at the same concurrency levels on NLD-8B.

Aggregate tok/sC = 248163264128
PDO1781303946606631719574808108
VGD159528274835741892271025810093

These results place the switching boundary between C = 4 and 8 for the GPU and workload used in our experiments. GPU compute-to-bandwidth ratios vary substantially, and acceptance length depends on the task. RecGuide therefore adapts the switching boundary to the hardware and workload, analogous to the task- and hardware-specific adaptation in DSpark's Hardware-Aware Prefix Scheduler.

Thank you for raising these questions about temperature sampling and strategy switching, which helped us extend the evaluation and clarify the concurrency boundary. We hope the responses above can address your concerns and contribute to a reconsideration of review score. Looking forward to discussing more with you.

Best,
Authors

AuthorsOpenReview ↗

Comment to Reviewer mopY

Dear Reviewer mopY,

Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above incorporating all requested experiments, analyses, and clarifications.

Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!

Reviewer_riE9OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to improve the self-speculation efficiency of joint AR-diffusion models like Nemotron-Labs-Diffusion under both low and high concurrency. At low concurrency, RecGuide uses the top-2 draft margin to predict likely rejections and performs extra drafting in parallel with the current verification. At high concurrency, it selects a per-request block size based on recent verifier entropy. Experiments on NLD-8B and NLD-14B across eight benchmarks in SGLang show faster single-request decoding than baseline methods.

Critical strengths
  1. This work targets an interesting problem. Self-speculation achieves long acceptance lengths with a unified draft/target model, but two full forward passes per round limit its actual speedup. The proposed method can reduce the number of forward passes for some blocks to one when draft positions are predicted correctly.

  2. The design choices are backed by measurements rather than chosen ad hoc. For example, the top-2 draft margin predicts verification failures well.

  3. The method is implemented in a real serving engine, and the evaluation covers two model sizes, eight benchmarks, concurrency levels from 1 to 128, and component ablations.

Critical weaknesses
  1. The major concern is that some important baselines are missing from the low-concurrency comparison. The proposed method essentially increases the verification width to boost TPF under low concurrency. This has been explored in prior works such as “PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding” and “Accelerating Speculative Decoding with Block Diffusion Draft Trees.” These works also draft and verify more tokens to boost TPF under low concurrency. The authors should benchmark their method against these approaches using the same verification width.

  2. Another concern is that the baselines at high concurrency are limited. Figure 5 only compares against LinearSS with fixed block sizes, with no AR, DFlash, or TiDAR curves. Thus, it is unclear whether self-speculation still beats plain AR or a lightweight drafter at C = 64 or 128, which is the question the introduction raises.

  3. The practical usability of the proposed framework as a unified serving system is questionable. The components for low and high concurrency closely follow prior work, such as draft-ahead from PEARL and SSD and the cost model from DSpark. More importantly, the two strategies are used in separate regimes (overlap only at C = 1 and adaptive sizing only at C ≥ 2), with no rule for switching between them as the load changes. In addition, the router requires offline profiling for each model and concurrency level.

  4. The clarity of the writing could be improved. For example, the joint drafting and verification process in Section 5.1 is difficult for a general audience to understand.

Questions for authors

In addition to the questions in the weakness section, I wonder whether DSpark's confidence head can be directly used to predict the block size under high concurrency.

3 comments
AuthorsOpenReview ↗

Response to Reviewer riE9: Part 1

Dear Reviewer riE9,

We sincerely thank you for your system-level perspective and for recognizing the technical soundness of our approach. We address your questions on baseline comparisons, related work, and strategy switching below.

1. Comparison with Low-concurrency Baselines

We agree that it is important to compare RecGuide with methods that improve speculative acceptance by expanding the verification width. We reproduced PRESTO and DDTree on NLD-8B and NLD-14B. For each benchmark, we set the baselines' verification widths to match RecGuide's measured average per-round computation on that benchmark. For PRESTO, we used the marginal-only scoring strategy reported in its NLD experiments. We follow the convention in Table 2 and report the average acceptance length (τ) per two-forward round, equivalent to one draft-verify round of standard self-speculation. RecGuide achieves the highest acceptance length on all eight benchmarks for both model sizes.

ModelMethodGSM8KMATH-500AIME25HumanEvalMBPPLCBGPQAIFEvalAvg.
NLD-8BPRESTO10.9512.2012.1510.879.939.0112.0410.7810.99
NLD-8BDDTree11.1312.2312.1010.959.949.0112.0410.8611.03
NLD-8BRecGuide13.9516.0815.6813.6411.0813.5414.8413.3314.02
NLD-14BPRESTO11.1011.8510.6813.0410.788.9712.049.6711.02
NLD-14BDDTree11.1011.8710.6813.0411.658.9912.059.7911.15
NLD-14BRecGuide13.6115.4313.4118.1912.5613.2716.2110.9714.21

We attribute this advantage to two properties. First, self-speculation with a shared backbone already produces high-quality drafts. At a matched verification width, it is more effective to branch at a few key positions than to allocate branches based on probabilities across the entire draft block. Second, RecGuide uses the extra computation to prepare drafts for the next round under candidate prefixes while verifying the current draft. When a candidate branch matches the verifier's result, the system proceeds directly to the next verification, removing a full-model forward from the critical path.

2. Comparison with High-concurrency Baselines

In the original manuscript, we focused the high-concurrency comparison on LinearSS and RecGuide because our cost analysis predicted that both would outperform DFlash and TiDAR in this regime. At a matched verification width, the verification stages of DFlash and LinearSS have the same computation and memory-access characteristics and therefore incur the same target-side forward cost at a given concurrency. Table 2 shows that LinearSS and RecGuide accept more than twice as many tokens per draft-verify round as DFlash on average. With the same backend, this difference in accepted tokens outweighs the benefit of DFlash's lightweight drafter at high concurrency.

TiDAR re-drafts from all possible positions in a block of B draft tokens, which introduces O(B²) computation. As Table 2 shows, this cost already limits the end-to-end benefit of its longer acceptance lengths at C = 1. The additional computation becomes more costly as concurrency increases. We now provide aggregate throughput measurements on NLD-8B at C = 64 and 128. The results confirm our analysis and show that RecGuide achieves the highest throughput at both loads.

MethodC = 64 (tok/s)C = 128 (tok/s)
AR56426499
DFlash69557402
TiDAR16251432
LinearSS78488162
RecGuide1025810093

The comparison with AR raises a further point. At sufficiently high concurrency, we expect AR to overtake LinearSS and eventually RecGuide as self-speculation becomes compute-bound. In practical cloud serving, long contexts consume substantial KV-cache capacity, limiting how far batch sizes can grow. This is why speculative decoding remains an attractive option for production serving workloads such as DeepSeek's.

AuthorsOpenReview ↗

Response to Reviewer riE9: Part 2

3. Comparison for Low-concurrency Strategy

Section 2 discusses how our draft-verify overlap differs from PEARL and SSD, and we will make this distinction more explicit. Both methods overlap a separate draft model with a target model. As discussed in their papers, they rely on deploying these models on different GPUs for parallel execution. When both models run on one GPU, they compete for memory bandwidth. This contention prevents overlap from delivering an end-to-end gain in the memory-bound, low-concurrency regime.

Consider the limiting case in which the draft and target models share an architecture, differ in parameter count, and have the same arithmetic intensity. Executing their forwards concurrently on one device offers no latency advantage over serial execution because both are limited by the same memory bandwidth. RecGuide is designed for self-speculation and supports overlap on a single GPU. Drafting and verification use the same backbone and model weights, so co-batching them allows the prospective drafts to use the GPU's spare compute during the memory-bound verification forward and translate that overlap into end-to-end speedup.

4. Comparison for High-concurrency Strategy & DSpark Confidence Head

RecGuide and DSpark both use hardware-aware cost modeling. DSpark adjusts draft confidence with its confidence head to prioritize current draft tokens for verification. RecGuide uses completed target-model verification to choose the next draft length. This has two advantages. First, predicting next-round acceptance from verification outcomes reduces the effect of discrepancies between draft and target logits. DSpark accounts for this discrepancy by adjusting its confidence-head outputs using an additional calibration set. Second, full acceptance identifies when to expand from a short block to a longer one. This transition requires a signal beyond draft confidence.

Using DSpark's confidence head to guide the next drafting budget is an interesting direction. Full DSpark training to obtain the head on NLD-8B and NLD-14B was beyond our compute budget, so we tested a trained Qwen3-4B DSpark model. To predict the next block size, we used either (a) the mean of the previous round's head outputs or (b) the first position where the output falls below a threshold.

Confidence-head signalPrecisionRecallF1
Mean output0.55120.56710.5590
First threshold crossing0.54280.56460.5535

These results can inform dynamic drafting in DFlash-based speculative decoding. Uniform guessing among three block sizes has an expected accuracy of 0.33. The confidence head performs above chance, with scores around 0.55, but still falls short of practical routing needs. This motivates a policy designed specifically for block-size switching.

5. Strategy Switching Across Concurrency Levels

Following DFlash and NLD, we reported single-request and batched results separately in the original submission. Because RecGuide uses different serving strategies at different loads, this presentation gave the impression that Predictive Draft-Verify Overlap (PDO) applies only at C = 1 and that the system must switch to Verification-Guided Adaptive Drafting (VGD) at C = 2.

We evaluated PDO and VGD at the same concurrency levels on NLD-8B.

Aggregate tok/sC = 248163264128
PDO1781303946606631719574808108
VGD159528274835741892271025810093

These results place the switching boundary between C = 4 and 8 for the GPU and workload used in our experiments. GPU compute-to-bandwidth ratios vary substantially, and acceptance length depends on the task. RecGuide therefore adapts the switching boundary to the hardware and workload, analogous to the task- and hardware-specific adaptation in DSpark's Hardware-Aware Prefix Scheduler.

6. Clarity of the Overlaped Draft-Verify Process

Thank you for pointing out that the joint drafting and verification process was difficult to follow. We have clarified the explanation in Section 5.1 and resubmitted the revised manuscript.

Thank you for your feedback on the baselines, related work, and presentation, which helped us strengthen the evaluation and clarify the manuscript. We hope the responses above can address your concerns and contribute to a reconsideration of review score. Looking forward to discussing more with you.

Best,
Authors

AuthorsOpenReview ↗

Comment to Reviewer riE9

Dear Reviewer riE9,

Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above incorporating all requested experiments, analyses, and clarifications.

Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request to facilitate reviewer discussion

Dear Area Chair and Reviewers,

Thank you for your time and feedback on our submission. We have posted point-by-point responses to all three reviews and updated the manuscript to address the comments.

As the author–reviewer discussion deadline approaches, we would be grateful if each reviewer could let us know whether our responses address their concerns or if any questions remain. We have yet to receive follow-up comments and would welcome the opportunity to clarify any outstanding points before the discussion closes.

We would also appreciate the Area Chair’s help in encouraging the reviewers to share their feedback while the discussion is still open.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.