- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RecGuide, a runtime orchestration framework for shared-backbone diffusion–AR self-speculation. At low concurrency, draft distributions predict likely verification failures and correction tokens, enabling prospective next-round drafts to be co-batched with current verification. At higher concurrency, historical verifier entropy, previous block size, and acceptance outcomes guide block-size selection, while bucketed execution reduces padding.
Critical strengths
In this paper, predictive correction and continuation branches allow next-round drafting to be co-batched with current verification, addressing the bottleneck of two sequential full-model forwards. End-to-end throughput measurements demonstrate practical benefits and reveal the importance of execution strategy.
Critical weaknesses
- The low- and high-concurrency mechanisms are evaluated separately at fixed loads, without demonstrating unified switching under dynamic loads. Moreover, a substantial portion of the Table 6 gain comes from bucketing, while simple adaptive policies using the same backend are absent. It is therefore difficult to separate the contributions of reciprocal prediction, execution optimization, and scheduling configuration to the claimed serving-frontier improvement.
- The lightweight-drafter comparison is not matched. DFlash is trained only on ShareGPT with the SpecForge recipe because of budget limits, whereas LinearSS, TiDAR, and RecGuide use the official NLD backbone.
Questions for authors
- With identical KV handling, kernels, and bucketing, how much improvement remains over a simple block expansion/contraction policy based on the previous acceptance length?
- After training DFlash to convergence on data comparable to the NLD drafter, does its single-request speedup remain clearly below RecGuide?
- Does throughput rise or fall when predictive overlap and adaptive block sizing are enabled together?
3 comments
Response to Reviewer hn6C: Part 2
3. Strategy Switching and Joint Execution
Following DFlash and NLD, we reported single-request and batched results separately in the original submission. Because RecGuide uses different serving strategies at different loads, this presentation gave the impression that Predictive Draft-Verify Overlap (PDO) applies only at C = 1 and that the system must switch to Verification-Guided Adaptive Drafting (VGD) at C = 2.
We compared PDO alone, VGD alone, and the two strategies together at the same concurrency levels on NLD-8B.
| Aggregate tok/s | C = 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|
| PDO | 1781 | 3039 | 4660 | 6631 | 7195 | 7480 | 8108 |
| VGD | 1595 | 2827 | 4835 | 7418 | 9227 | 10258 | 10093 |
| Joint | 1796 | 2936 | 4569 | 6291 | 7395 | 8216 | 7230 |
Joint execution offers little benefit over PDO at low load. It is slightly ahead of PDO at C = 2 and falls behind from C = 4 through C = 16. In this range, VGD's relatively conservative block-size choices give up potential acceptance opportunities that PDO could exploit. At C = 32 and 64, VGD offsets some of PDO's additional computation. VGD alone still leads the combination. At the highest concurrency, C = 128, the combination is slower than either strategy alone. This shows how strongly the two strategies can interfere when both draw on the same compute budget, given that each was designed for a different load regime.
These results place the switching boundary between C = 4 and 8 for the GPU and workload used in our experiments. GPU compute-to-bandwidth ratios vary substantially, and acceptance length depends on the task. RecGuide therefore adapts the switching boundary to the hardware and workload, analogous to the task- and hardware-specific adaptation in DSpark's Hardware-Aware Prefix Scheduler.
Thank you for your questions about throughput gains, DFlash training, and joint execution, which helped us strengthen the comparisons and clarify the role of each strategy. We hope the responses above can address your concerns and contribute to a reconsideration of review score. Looking forward to discussing more with you.
Best,
Authors
Comment to Reviewer hn6C
Dear Reviewer hn6C,
Thank you again for your thoughtful review and constructive feedback. As the discussion period closes shortly, we wanted to send a brief follow-up to note that we have posted our detailed response above incorporating all requested experiments, analyses, and clarifications.
Please let us know if you have any remaining questions before the window closes---we would be glad to clarify anything further. Thank you again for your time and consideration!
Response to Reviewer hn6C: Part 1
Dear Reviewer hn6C,
We sincerely thank you for highlighting the practical benefits of co-batched drafting and verification. We also appreciate your suggestions regarding the routing comparison, DFlash reproduction, and joint execution. Below are our responses to your concerns.
1. Fair Comparison with Simple Block-switching Strategies
We agree that comparing with simpler block-size policies under identical KV management, kernels, and bucketing is important. On NLD-8B, we considered three additional baselines. (a) Direct chooses the block-size bin from the previous round's acceptance length. (b) Direct + Full Signal uses the same direct mapping and adds the previous round's full-acceptance indicator. (c) Calibrated + Full Signal uses the same calibration set as RecGuide to calibrate the mapping from the previous acceptance length to a block-size bin and also uses the full-acceptance indicator. The table reports the average throughput change relative to fixed-block serving across the tested concurrency levels.
The two uncalibrated policies have difficulty adapting to fluctuations in acceptance length during decoding. They yield little or negative throughput gain. The full-acceptance signal helps the router expand short blocks when the whole draft is accepted. Using the calibration set to map acceptance length to the next block size further improves throughput. RecGuide still leads the strongest calibrated baseline by 5.02 percentage points with the same backend.
We emphasize that the large gain after adding bucketing in Table 6 is a joint gain from adaptive drafting and bucketing. Adaptive drafting saves computation by choosing request-specific draft lengths. Within a mixed-length batch, shorter drafts would be padded to the longest block, eroding those savings. Bucketing groups requests by block size and preserves the savings from the routing policy, making it essential for adaptive drafting to deliver end-to-end throughput gains.
2. Training Data and Convergence of the Reproduced DFlash
We compared the sample and token counts of the training data that Z-Lab released for DFlash, the ShareGPT data used in our reproduction, and the training data described in the NLD report. For DFlash, we count tokens over six epochs. Our training configuration follows the official recipe of SpecForge in the SGLang community, and the amount of training data is on the same scale as Z-Lab's release.
Matching NLD's training data presents two challenges. First, the scale is large. The NLD report describes training on 256 NVIDIA H100 GPUs. Second, those 300B tokens are used for joint AR and diffusion training of the shared backbone. NLD uses that backbone for drafting as well as verification. DFlash trains a lightweight drafter separately from the target model, at the smaller scale needed for a dedicated drafter.
To check how much further the reproduced drafter could improve with training, we measured acceptance length after each epoch on NLD-8B.
The results show a clear convergence trend. RecGuide delivers approximately 1.5× the single-request end-to-end throughput of our DFlash reproduction. Given the convergence trend and the size of this gap, we expect that additional epochs or a larger dataset such as Open-PerfectBlend would still leave DFlash well below RecGuide.