acceptodds
Under review as a conference paper at ICLR 2027

CHARD: Efficient Block-Diffusion Speculative Decoding via Chunked Verification and Immediate Redrafting

Abstract

Block-diffusion drafters generate multiple consecutive draft tokens in a single forward pass and often yield long accepted prefixes. In batched serving, however, requests within a physical target batch typically share a common verification width. This coupling creates two inefficiencies. First, when a request rejects before the end of the verification window, target computation on the remaining suffix cannot contribute to its output. Second, accepted progress varies substantially across concurrent requests and workloads. Although requests undergo the same fixed-width verification round, low-acceptance requests commit fewer tokens, reducing their token production rate and extending the tail of time per output token (TPOT). Consequently, long accepted prefixes do not fully translate into serving gains. We present CHARD (Chunk-Aware Redrafting and Decoding), a lossless serving system that decouples proposal horizon, verification granularity, and request recovery. Chunked verification consumes a long proposal in bounded-width chunks and reuses the remaining suffix only when the current chunk is fully accepted and the target-produced boundary token matches the stored draft anchor. This avoids target computation on invalidated suffixes. Immediate redrafting resolves request state after every chunk: a terminated request becomes eligible for a replacement proposal at the next scheduling opportunity, while requests with valid suffixes continue verification. We implement CHARD in SGLang and evaluate it on Qwen3-8B and Qwen3.5-27B with their corresponding DFlash drafters. For 16-token proposals, CHARD improves throughput by up to 30.4% and reduces request-level p99 TPOT by up to 27.4% relative to one-shot DFlash with the same proposal horizon. CHARD retains long draft proposals, requires no learned controller, and preserves the target model's deterministic decoding semantics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_By1ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CHARD modifies the proposal lifecycle in block-parallel speculative decoding to provide finer-grained target feedback and request-local recovery. Instead of verifying a full DFlash proposal in a single target call, it verifies the proposal in smaller chunks. Requests that pass a chunk retain the unverified proposal suffix, while requests that encounter a rejection can immediately redraft without waiting for other requests to finish consuming their proposals. The authors implement CHARD in SGLang and provide sufficient conditions under which this scheduling preserves the target distribution.

On Qwen3-8B and Qwen3.5-27B, CHARD reports up to 30.4% higher throughput and 27.4% lower request-level p99 TPOT than one-shot DFlash B16. An ablation comparing chunk-only verification with full CHARD shows that immediate redrafting provides additional throughput and latency gains beyond chunking alone.

Critical strengths
  1. CHARD is implemented as an end-to-end serving mechanism rather than evaluated only through simulation or trace replay. The SGLang implementation maintains per-request proposal queues, retains valid suffixes across target calls, and allows replacement proposals to re-enter subsequent verification batches. It also handles the state and KV-cache management required for this asynchronous proposal lifecycle.

  2. Table 2 cleanly separates the benefit of request-local recovery from chunked verification itself. On Qwen3-8B at BS32, throughput is 3,778 tokens/s with one-shot B8, 3,666 with chunk-only verification, and 3,971 with full CHARD. The two chunked variants have essentially identical evaluated positions and target calls per proposal, making the throughput difference a convincing isolation of the benefit of immediate redrafting rather than reduced verification work.

Critical weaknesses
  1. The incremental benefit over adaptive verification is unclear. D-Cut and DSpark already address heterogeneous acceptance through adaptive verification. Table 2 isolates the benefit of CHARD's request-local recovery over fixed-width chunking, but the paper does not show whether this benefit remains when an existing adaptive verification policy is enabled. A comparison that adds CHARD-style suffix retention and recovery to an adaptive policy would clarify whether CHARD is complementary to, rather than substitutable for, prior methods.

  2. The D-Cut comparison is not sufficiently controlled. D-Cut B8 with a 50% verification budget is evaluated in vLLM, whereas CHARD B8/C4 runs in SGLang. The D-Cut results are also averaged across concurrency settings while the text highlights a BS4 CHARD result. Thus, the reported numbers do not provide a matched head-to-head comparison. A same-runtime comparison using D-Cut's adaptive policy is needed to establish the relative benefit of CHARD.

  3. The DSpark experiment does not compare against DSpark's verification scheduler. The experiment uses the DSpark draft head but compares one-shot verification with CHARD, rather than DSpark's confidence-scheduled verification. It therefore demonstrates compatibility with the DSpark drafter, not an advantage over DSpark's scheduling policy. Moreover, at BS32, CHARD improves low-acceptance median TPOT from 8.82 to 8.52 ms but worsens high-acceptance TPOT from 5.84 to 6.67 ms; this trade-off deserves further analysis.

  4. The choice of chunk width C=4 is insufficiently justified. The paper states that a model-level sweep selected C=4, but does not report the sweep or its sensitivity to proposal width and concurrency. Figure 3 only estimates logical verification work and does not capture the runtime overhead of additional target calls. Since this trade-off is central to CHARD, end-to-end results across different C values are needed.

  5. The serving workload is overly controlled. The evaluation uses fixed concurrency, decode-only execution, fixed 1,024-token outputs, and no early EOS. There is no arrival-driven workload with variable request lengths, despite CHARD's motivation being request-level recovery under heterogeneous progress. In addition, the reported p99 TPOT is a percentile of per-request average TPOT rather than individual inter-token gaps, so transient token-delivery stalls are not characterized.

Questions for authors
  1. Can the authors compare D-Cut and CHARD under the same serving runtime, model, drafter, GPU, prompts, and concurrency settings? Please evaluate D-Cut with its native adaptive verification policy rather than only the reported 50% budget configuration, and report throughput and latency separately for each concurrency level.

  2. Why does the DSpark-head experiment compare CHARD against one-shot verification rather than DSpark's confidence-scheduled verification? Does CHARD still provide additional benefit when DSpark's verification policy is enabled? A combined DSpark scheduling + CHARD recovery experiment would clarify their complementarity.

  3. What were the results of the sweep used to select C=4? Please report throughput, TPOT, and target-call counts across several C values for both B8 and B16.

  4. How does CHARD's scheduling and redrafting affect high-acceptance requests? In particular, what causes the increase in their median TPOT in the DSpark experiment, and is this a fundamental trade-off of request-local recovery or an artifact of the current scheduler?

  5. How does CHARD perform under more realistic online serving conditions, including request arrivals, variable output lengths, and prefill/decode interference? It would also be useful to report a tail metric over individual inter-token gaps, separately from the p99 of request-average TPOT.

3 comments

Response to Reviewer By1Z

We thank Reviewer By1Z for the detailed comments. We first state the system positioning of this paper, then respond point by point.

System positioning

Existing block-diffusion serving binds the proposal width B, the target verification granularity C, and the request recovery timing into a single static configuration. This configuration is not the same optimum under different concurrency. On the DFlash head, the crossover is as follows:

  • A fixed B8 is overtaken by B4 at BS16/32, where B4 is higher by 4.3%/7.4%;
  • A fixed B4 is 37.2% below B8 at N1;
  • A fixed B16 is below B8 at every deployment point, by 19.3% at BS16 and 22.7% at BS32.

The same crossover appears on the DSpark head with different absolute magnitudes (see the calibration table in Q5.2): the optimum is B8 at N1–8 and B4 at N16–32, where a fixed B4 loses 23.6%–70.0% at low concurrency and a fixed B8 loses 1.2%–3.9% at high concurrency. What is invariant across the two heads is the direction of the crossover, not its size.

The system in this paper is not a single-point speedup on a fixed block, but an adaptive serving system: it decouples B, C, and the recovery timing, and selects the configuration according to concurrency. Concretely, for the hardware and workloads in this paper, the calibration result is B8 at low concurrency and B8/C4 at high concurrency; on different hardware or workloads the calibration table yields a different per-concurrency configuration, while the mechanism is unchanged.

The system is not tied to DFlash: on MTP 7/1/8, MTP+CHARD matches or beats the strongest MTP at every concurrency point.

Because absolute numbers depend on the draft head, we label the head wherever a result is head-specific: Q1 is measured on DFlash, and Q2 and Q5 on DSpark. Conclusions about the mechanism hold for both.


Q1. D-Cut vs. CHARD under identical runtime, model, drafter, GPU, prompts, and concurrency

We reproduced D-Cut in SGLang and compared D-Cut (auto) against CHARD at the B8 configuration, using the same DFlash head, a single H20, Qwen3 8B as the target, the same six workloads, and a sweep over concurrency:

ConcurrencyD-Cut auto (B8)CHARD policyCHARD gain
1513.5632.0+23.1%
2984.31191.2+21.0%
41782.32123.3+19.1%
82691.33036.0+12.8%
163266.33547.3+8.6%
323753.83943.0+5.0%

Q2. Relation between the DSpark confidence scheduler and CHARD

DSpark and CHARD share the goal of reducing verification computation, but they target different things:

  • DSpark performs confidence pruning on the draft-generation side, deciding which draft tokens enter verification;
  • CHARD changes verification granularity and recovery timing on the draft-consumption side, deciding how already-verified chunks are committed and how the unverified suffix is retained.

432-point comparison (same H20, same DSpark head, same prompts, six workloads, N1–32, three runs):

ConcurrencyDSpark + dynamicCHARDCHARD vs. dynamic
1769789+2.6%
213731428+4.0%
423612485+5.3%
834973603+3.0%
1641384169+0.7%
3243604458+2.3%

The two are currently not complementary: CHARD alone already exceeds DSpark+dynamic by 0.7%–5.3%. The reason is that they act on different stages of verification computation: DSpark reduces the tokens that enter verification, whereas CHARD changes how those tokens are consumed.

Because the stages differ, stacking the two is possible in principle: DSpark first prunes draft generation, then CHARD consumes the result in chunks. We leave this combination to future work.

Q3. Justification for C=4: C sweep at B8/B16

BS32 (macro-average over six workloads, tok/s / p99 TPOT ms):

ProposalC3C4C5C6C7C8
B83789 / 8.313971 / 7.963773 / 8.433856 / 8.303741 / 8.563778 / 8.49
B163700 / 8.543755 / 8.463718 / 8.613616 / 8.873563 / 8.953550 / 9.05

BS16:

ProposalC3C4C5C6C7C8
B83221 / 4.973439 / 4.683395 / 4.803454 / 4.743272 / 5.053383 / 4.95
B163096 / 5.213307 / 4.933299 / 4.993266 / 5.083131 / 5.273233 / 5.17
  • BS32: both B8 and B16 peak at C4; B8 is 5.1% above C8, and B16 is 5.8% above C8.
  • BS16: B16 peaks at C4; for B8, C4 and C6 are essentially tied (0.4%).
  • C4 is not a globally fixed configuration; on the current hardware, low concurrency selects B8 (C=B) and high concurrency selects B8/C4.

Target calls per proposal:

  • B8/C4 1.59, B8/C3 1.69, B8/C8 1.00;
  • B16/C4 2.21, B16/C3 2.76, B16/C8 1.00.

C4 sits above C8 and below C3 in call count; it reaches the peak between "earlier recovery" and "additional calls."

Q4. TPOT regression for high-acceptance requests

This is not a harmful regression. What lowers SLO is the slow requests, not the high-acceptance requests that are already fast. The idea of this paper is to give up part of the fast requests' performance in exchange for the efficiency of the slow requests.

Under uniform C4, the median TPOT of high-acceptance requests rises from 5.84 to 6.67 ms; under the same mechanism:

  • Qwen3-8B: TPOT of the lowest-acceptance quartile −26.0%, highest-acceptance quartile +1.9%;
  • Qwen3.5-27B: lowest-acceptance quartile −27.7%, highest-acceptance quartile +5.9%.

The high-acceptance group pays about 2%–6% in TPOT, while the low-acceptance group gains about 26%–28%. SLO is determined by the latter, so the exchange is favorable. The 5.84→6.67 ms figure is the median on the DSpark workload; the bucketed mean over six workloads is +1.9%, consistent in direction.

Q5. More realistic online serving

1. Online conditions ultimately all act on instantaneous concurrency

Request arrivals, variable output lengths, and PD disaggregation all affect the decode side through one quantity: the number of active decode requests currently participating in model calls, N.

  • High arrival rate → N increases; low arrival rate → N decreases.
  • Long outputs keep requests in the decode pool longer → N increases at the same arrival rate; short outputs → N decreases.
  • PD disaggregation changes the resource ratio between P and D, but N on the decode side is still determined by queueing in the D pool.

The system does not need to predict the arrival trace, nor does it need to know the output-length distribution; it reads N at runtime and looks up the optimal configuration for that N in the calibration table.

2. Calibration table: the optimal configuration for each N

DSpark-head calibration (432 points: six workloads, three runs, N1–32). The table below shows two representative configurations on Qwen3 8B on H20; changing hardware or model requires regenerating this calibration table:

ConcurrencyB8 optimizedB4 nativeBest at this concurrencyvs fixed B8vs fixed B4
1789464B80+70.0%
21428889B80+60.6%
424851652B80+50.4%
836032915B80+23.6%
1641694220B4+1.2%0
3244584632B4+3.9%0
  • The optimum at N1–8 is B8; a fixed B4 loses 23.6%–70.0% at these points.
  • The optimum at N16–32 is B4; a fixed B8 loses 1.2%–3.9% at these points.
  • Per-concurrency selection takes the larger of the two at every N and is therefore never worse than either fixed configuration.

We note explicitly that this table is measured on the DSpark head, whereas the crossover figures quoted in the "System positioning" section are measured on DFlash (B4 higher by 4.3%/7.4% at BS16/32; B4 37.2% below B8 at N1; B16 below B8 by 19.3%/22.7%). The magnitudes differ because the two heads have different acceptance profiles and draft costs, which shifts where, and how sharply, the crossover occurs. The invariant across both heads is the crossover itself — B8 wins at low concurrency, B4 wins at high concurrency — and hence the need for per-concurrency selection. Our claims rest on that direction, not on any single magnitude.

3. Request arrivals

The arrival process determines the time series of N. The selector reads the current N at every scheduling step and looks up the configuration:

  • Burst arrivals: N rises quickly and the selector switches to the high-concurrency configuration;
  • Low arrival rate: N falls and the selector returns to B8, avoiding the loss of the high-concurrency configuration under low load.

The selector is reactive; it needs no arrival prediction, only the current value of N.

4. Variable output length

Output length affects N through residence time:

  • Long outputs: requests stay in the decode pool longer, so N is larger at the same arrival rate, and the selector favors the high-N configuration;
  • Short outputs: requests leave quickly, N is smaller, and the selector favors the low-N configuration.

The calibration table covers N1–32, and both directions fall inside it. The output-length distribution does not need separate modeling; its effect is already absorbed by N.

5. PD disaggregation

CHARD is a pure decode optimization and does not change the P rate.

  • With the P rate unchanged and decode faster, the bottleneck shifts to P and the decode side may idle.
  • This is a resource-ratio problem: adding P nodes or adjusting the P:D ratio converts the decode savings into system throughput.
  • N on the decode side is still determined by queueing in the D pool, so the calibration table continues to apply; after scaling out P, the system can admit more requests, the operating range of N shifts upward, and the selector still picks configurations by N.

6. Individual inter-token gap reported separately from request-average TPOT

  • Individual gap p99: the p99 over all single-token intervals;
  • Request-average p99: first compute each request's mean commit gap, then take the p99 over all requests.
ConcurrencyCHARD gap p99CHARD req-avg p99B8 gap p99B8 req-avg p99gap diffreq-avg diff
18.738.578.738.570%0%
29.069.169.069.160%0%
427.4010.2427.4010.240%0%
832.4714.3032.4714.300%0%
1642.1815.2656.5624.98−25.4%−38.9%
3265.8925.9199.8543.79−34.0%−40.8%
Reviewer_PYpqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a novel scheduling approach to improve the serving throughput and request-level P99 TPOT of existing speculative decoding approaches. Unlike conventional approaches, they chunk the verification stage into smaller block sizes and terminate verification early at a request level upon detecting a rejection. Requests with rejected proposals can immediately become eligible for redrafting, while requests with accepted prefixes can continue verifying remaining draft positions. A request-aware scheduler dynamically batches verification and drafting operations across requests, allowing them to progress independently, reducing wasted verification computation and delays in recovering from rejected proposals. The authors implement CHARD on top of DFlash, and evaluate it using Qwen3-8B and Qwen3.5-27B models across various tasks. Compared with DFlash using a draft block size of 16, CHARD reports up to 30.4% higher throughput and 27.4% lower p99 TPOT. The paper also presents ablations demonstrating the contribution of immediate redrafting beyond chunked verification alone, as well as experiments on workloads with heterogeneous acceptance rates.

Critical strengths
  1. Simplicity of approach: Chunked verification offers a simple and potentially effective way to reduce redundant verification computation on rejected tokens.

  2. Critical observation: The authors demonstrate that immediate redrafting is important for achieving low p99 request-level TPOT, as chunked verification alone can introduce substantial overhead from additional target calls. Authors back it up with thorough ablation studies.

  3. Evaluation across models and tasks: The authors report results across two model sizes (Qwen3-8B and Qwen3.5-27B) and diverse tasks, including mathematical reasoning, coding, and instruction following.

Critical weaknesses
  1. Limited gains over stronger baselines: The headline 20–30% throughput improvements are primarily against DFlash B16, which consistently underperforms DFlash B8. Compared to DFlash B8, CHARD achieves only 1.6–5.1% throughput improvement. Moreover, CHARD B16/C4 consistently underperforms CHARD B8/C4, raising questions about the practical significance of the headline results.

  2. Insufficient evaluation across serving regimes: The main experiments focus on batch sizes 16 and 32, while the smaller-batch study (Table 3) only compares against DFlash B4. Chunked verification introduces additional target-model calls, which could be particularly expensive at low concurrency.

Questions for authors
  1. Given that DFlash B8 consistently outperforms DFlash B16, and CHARD B8/C4 outperforms CHARD B16/C4, what is the practical motivation for emphasizing B16 results? Can the authors demonstrate a regime where larger proposal blocks provide an advantage over the best-performing B8 configuration?

  2. Could the authors provide comparisons against DFlash B8/B16 at smaller batch sizes (1–8)? Since chunked verification increases the number of target-model invocations, how does its effectiveness vary between memory-bandwidth-bound low-concurrency inference and higher-concurrency serving?

  3. The authors demonstrate CHARD's compatibility with a DSpark draft head, but how does CHARD compare against DSpark's original confidence-based verification-length scheduling? Can the authors provide a comparison against this adaptive baseline using the same inference engine and hardware?

  4. The evaluation is limited to dense target models. How would CHARD's benefits change for MoE targets, particularly under distributed serving configurations where verification costs and communication overhead may behave differently?

2 comments

Response to Reviewer PYpq

Clarification: What CHARD Is

Existing block-diffusion serving systems bind three decisions into a single static configuration: proposal width B, target verification granularity C, and request recovery timing. No single static configuration is optimal across concurrency: low concurrency favors a long block, while high concurrency favors a short block. A fixed B8 loses 3.9% at high concurrency, a fixed B4 loses 70.0% at low concurrency, and a fixed B16 is below B8 at every deployment point. CHARD is therefore not a single-point speedup at a fixed block; it is a dynamic-concurrency-adaptive serving system that decouples B, C, and recovery, selects the configuration for the current concurrency, and approaches the per-concurrency optimum.

The system keeps a B-wide proposal as request-local state and consumes it in C-wide target-confirmed chunks. A continuing request retains its valid suffix; a terminated request becomes eligible to redraft at the chunk boundary and rejoin a later verification opportunity. We implement the scheduler, KV commit/rollback, CUDA Graph, and attention-metadata support in SGLang, and establish the lossless conditions under which incremental consumption preserves the target distribution. The system is not tied to DFlash: it also applies to the MTP head, where MTP+CHARD matches or exceeds the strongest MTP at every concurrency.

Given this system framing, we accept the reviewer's central criticism: B16 should not be the main result. B16 is below B8 at every deployment point, and CHARD B16/C4 is below B8/C4. The revision removes the 30.4% B16 headline, makes B8 the primary proposal width, and keeps B16 only as a same-width control. The recommended policy is B8/C8 for BS1-8 and B8/C4 for BS16/32. We answer Q1-Q4 below.


Q1. Why emphasize B16? Is there a regime where a larger block beats B8?

A1. We agree with the reviewer. B16 relative to B8 is:

DeploymentB16B8B16 vs. B8
8B BS1627283383-19.4%
8B BS3229223778-22.7%
27B BS1622632884-21.5%
27B BS3225203376-25.4%

CHARD B16/C4 is also 3.8%-5.8% below CHARD B8/C4. We have not measured a regime where B16 beats B8.

B16 serves only as a same-width control: it fixes the proposal width to isolate the effect of chunked verification. It is not a recommended deployment configuration. The original draft placed B16 in the main-result position; the revision removes the B16 headline, makes B8 the main table, and keeps B16 only as a same-width control. The deployment policy is B8 for BS1-8 and B8/C4 for BS16/32.

Q2. Comparisons against DFlash B8/B16 at small batch sizes (1-8); does chunking help at low concurrency?

A2. Low concurrency is memory-bandwidth-bound, and chunked verification adds target calls. The optimal configuration at BS1-8 is therefore B8, not B8/C4. DFlash measurements (Qwen3-8B TP1, six-workload average, three repetitions):

BSDFlash B8CHARD B8/C4Choice
1632418B8/C8
21191800B8/C8
421231483B8/C8
830362525B8/C8
1634723567B8/C4
3237463982B8/C4

This table answers the reviewer's question directly:

  1. BS1-8: B8 is optimal. CHARD B8/C4 is 16.8%-33.9% below B8; the system selects B8/C8.
  2. BS16-32: B8/C4 is optimal. B8/C4 is +2.7%/+6.3% above B8; the system selects B8/C4.

The data supports the per-concurrency policy: B8 at low concurrency, B8/C4 at high concurrency.


Q3. Comparison with DSpark's native confidence-based verification-length scheduling

A3. We add an end-to-end comparison on the same hardware (H20), the same DSpark head, the same prompts, six workloads, and N1-32:

ConcurrencyDSpark + dynamicDSpark + CHARDCHARD vs. dynamic
1769789+2.6%
213731428+4.0%
423612485+5.3%
834973603+3.0%
1641384169+0.7%
3243604458+2.3%

CHARD is 0.7%-5.3% above the native dynamic scheduler.


Q4. MoE targets and distributed serving

A4. The Qwen3 results are on dense models: low concurrency uses B8/C8, and mid/high concurrency uses B8/C4. For MoE, splitting a chunk reduces the expert weight access volume per call. If the acceptance length is similar, each chunked MoE call is faster, so CHARD is expected to be more beneficial on MoE than on dense models.

However, the implementation complexity on MoE is significantly higher than on dense models:

  1. Expert routing and all-to-all: each chunk contains a different token set, so expert load and communication volume vary across chunks; chunk boundaries must be synchronized across ranks, and the dense per-request split cannot be reused directly.
  2. Dynamic shapes and CUDA Graph: changing C changes the shapes of expert GEMMs and attention metadata; each C needs its own captured or selected graph, and EP/TP requires communication splitting and overlap.
  3. KV and request state: chunk-level commit/rollback must remain consistent across EP/TP ranks; redrafting couples with expert-parallel scheduling.
  4. Scheduler: chunk-level recovery interacts with expert-parallel batch formation and all-to-all overlap; the dense scheduling path cannot be transferred directly to MoE.

Therefore, an end-to-end measurement on DeepSeek V4 Flash requires the system changes above; it is not a simple model swap. Due to time constraints, we have not yet completed the TP/EP measurements on this model. We will add DeepSeek V4 Flash results in the revision.

Reviewer_3W83OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to improve the serving efficiency of block speculative decoding methods such as DFlash. It finds that a request with low acceptance must wait for verification of the entire batch before it can redraft. It proposes CHARD, which maintains a wide B-token proposal but verifies it in shorter C-token chunks, allowing requests that finish verification to redraft immediately and join the next verification call. Experiments in SGLang on Qwen3-8B and Qwen3.5-27B across six workloads show improvements in throughput and TPOT.

Critical strengths
  1. This work studies a practical problem. Block-diffusion drafters are attractive for speculative decoding, but their performance under batched serving is less promising due to the extra computational cost. This work addresses this issue from a serving perspective.

  2. Separating the proposal width from the verification granularity is a simple and well-motivated idea.

  3. The method is implemented in SGLang, and the evaluation covers two model scales, six workloads, and two concurrency levels.

Critical weaknesses
  1. The first concern is that the gains over simple alternatives are small. The headline 30.4% gain is measured against one-shot DFlash B16, which is slower than DFlash B8 at every deployment point. Against B8, the gain drops to 1.6–5.1%, and simply shrinking the block to B4 gives nearly the same throughput at BS16/32 (see Table 3).

  2. Another major concern is that the throughput and speedups of all baselines are low. The speedups in Table 1 range from 1× to 1.61×, which are far lower than those reported in the DFlash and DSpark papers. Also, on Qwen3.5-27B, the native MTP baseline outperforms CHARD in both throughput and TPOT. These two points raise questions about how promising the speedup offered by the proposed method is.

  3. The technical novelty and insights are limited. The proposed method is an inference implementation trick that also introduces extra complexity and potential inefficiencies. Specifically, it splits one verification process into several chunks, potentially lowering utilization, while independent redrafting may further reduce utilization. It is unclear whether this extra complexity pays off.

  4. The writing clarity of the whole paper can be improved.

Questions for authors

My questions have been provided in the weakness section.

3 comments

Response to Reviewer 3W83

We thank the reviewer. We accept the three core criticisms: the B16 main result used the wrong baseline (B16), B4 merely ties CHARD at high concurrency, and the 27B MTP baseline is stronger. The revision will reframe the narrative from "two local problems" to "existing methods cannot adapt to dynamic concurrency," and will promote the per-concurrency configuration envelope to the main result. Below we first clarify the system positioning, then respond to Q1–Q4.

Clarification before the response: the system positioning of this paper

Existing block-diffusion serving binds three decisions into a single static configuration: the proposal width B, the target verification granularity C, and the request recovery timing. This static configuration is not the same optimum at low and high concurrency: low concurrency needs long blocks to increase acceptance length, whereas high concurrency needs short blocks to reduce target computation. A fixed B8 loses 3.9% at high concurrency, a fixed B4 loses 70.0% at low concurrency, and a fixed B16 is below B8 at every deployment point. This paper is therefore not a single-point speedup on one fixed block, but a dynamically concurrency-adaptive serving system: it decouples B, C, and recovery, selects a configuration according to the current concurrency, and keeps the system close to the per-concurrency optimum across the whole concurrency range.

Concretely, the system keeps the B-wide proposal as request-local state and consumes it incrementally in C-wide target-confirmed chunks; continuing requests retain their valid suffix, while terminated requests are re-drafted immediately at the chunk boundary and join subsequent verification. We implement scheduling, KV commit/rollback, CUDA Graph, and attention metadata support in SGLang, and we give the lossless condition under which incremental consumption preserves the target distribution. The system is not tied to DFlash: it is equally effective on MTP 7/1/8, and MTP+CHARD matches or beats the strongest MTP at every concurrency point.

Based on this positioning, we accept the reviewer's three specific criticisms and revise the main results accordingly:

  • The 30.4% B16 main result used the wrong baseline and has been removed;
  • The gain on B8 is +1.6%–5.1%; B4 ties CHARD at BS16/32, with −1.0% at BS32, and we do not claim a lead at high concurrency;
  • On 27B the strongest baseline is MTP; MTP+CHARD matches or beats the strongest MTP, and we report MTP+CHARD as the best configuration on 27B.

At the same time, the paper's narrative changes from "wasted suffix computation + acceptance-rate heterogeneity" as two local problems to "static configurations cannot adapt to dynamic concurrency" as one system problem; the original two problems become two manifestations of that problem.

Q1. The gain over a simple alternative is small

A1. We agree. The 30.4% is a single-point result of the B16 family on Qwen3.5-27B/BS32; B16 is slower than B8 at every deployment point (2728<3383, 2922<3778, 2263<2884, 2520<3376). This number has been removed from the abstract and the contributions and is no longer used as a main result.

The correct same-width comparison on B8 is: Qwen3-8B +1.7% / +5.1%, Qwen3.5-27B +1.6% / +3.3% (BS16/32). For B4 on the coding workload: BS1/2/4/8 are +5.5% / +3.9% / +2.6% / +1.0%, BS16 +0.8%, BS32 −1.0%.

Per-concurrency results for B8/B4 (DSpark head, six workloads, three runs):

ConcurrencyB8 optimizedB4 nativeBest at this concurrencyvs fixed B8vs fixed B4
1789464B80+70.0%
21428889B80+60.6%
424851652B80+50.4%
836032915B80+23.6%
1641694220B4+1.2%0
3244584632B4+3.9%0

A fixed B8 loses 3.9% at N32, and a fixed B4 loses 70.0% at N1. Only by selecting B8 or B4 according to concurrency can the system reach the measured optimum at every point. DFlash shows the same crossover: B8/C4 for BS1–16 and B4 for BS32. B16 is retained only as a same-width family result and is no longer used as the main baseline.

Q2. Baseline throughput is low; 27B MTP is stronger

A2. We answer in three parts.

(1) Hardware and roofline

All experiments in this paper were run on NVIDIA H20 (96 GB HBM3, 4 TB/s, BF16 148 TFLOPS). DFlash's main experiments were run on H200 (BF16 989 TFLOPS, 5 TB/s), and the SGLang throughput experiments on a single B200 (BF16 2250 TFLOPS, 8 TB/s); DSpark's public results are on H100 80GB (BF16 989 TFLOPS, 3 TB/s).

GPUBF16 (TFLOPS)Bandwidth (TB/s)Roofline ridgeUse
H20148437This paper
H1009893295DSpark
H2009895206DFlash main experiments
B20022508281DFlash SGLang

H20's BF16 compute is 15.0% of H100/H200 and 6.6% of B200; its bandwidth is only 80.0% / 50.0% of H200/B200. H20's ridge point is only 37, whereas H100/H200/B200 are 295/206/281.

Autoregressive decoding at BS32 has an arithmetic intensity of about 32 FLOP/byte; H20 already reaches 86.5% of its ridge, while H100/H200/B200 reach only 10.8% / 15.5% / 11.4%. Target verification at B=8 has an AI of about 256 FLOP/byte, i.e., 6.9× H20's ridge, and is strongly compute-bound. 3971 tok/s at BS32 on 8B corresponds to roughly 65 TFLOPS of target GEMM, 44.0% of H20's BF16 peak; the same 3971 tok/s is only 6.6% on H200 and 2.9% on B200. This counts only target GEMM, excluding draft and attention. H20 already reaches 86.5% of its ridge for AR at BS32, and becomes compute-bound once draft and verification are included; H200/B200 still have 84%–89% of compute headroom. We report absolute throughput and roofline utilization on H20 and do not claim cross-hardware speedups.

(2) MTP's advantage comes from head quality and draft cost, not from CHARD's consumption mechanism

The DFlash head is an external 5-layer draft trained on public open-source mixed data; MTP is the native head of Qwen3.5-27B, jointly trained with the target and aligned with the target's training data. Per-position acceptance rates are as follows:

HeadProposal widthτPer-position acceptance rate
DFlash B885.6270.3%
DFlash B16167.5947.4%
MTP 3/1/443.6190.3%

DFlash's absolute τ is higher (5.62 vs. 3.61), but that is an 8-token proposal against a 4-token proposal; the per-position acceptance rate is only 70.3% for DFlash versus 90.3% for MTP. MTP achieves higher throughput with a shorter proposal, a higher per-position hit rate, and lower draft cost. This gap stems mainly from head quality and draft cost, not from CHARD's consumption mechanism; the evidence is that CHARD applied to MTP gives the results below.

(3) CHARD combined with MTP

On Qwen3.5-27B TP4, with MTP 3/1/4 as the baseline:

TP4 configurationBS8BS16BS32
MTP 3/1/4 (baseline)213931173795
MTP 7/1/8 (one-shot)210427343342
MTP 7/1/8 + CHARD221632043785
vs 3/1/4+3.6%+2.8%−0.3%
vs 7/1/8+5.3%+17.2%+13.3%

At BS8/BS16, MTP 7/1/8 + CHARD is +3.6% / +2.8% above the 3/1/4 baseline; at BS32 it is −0.3%, i.e., a tie. Relative to 7/1/8 without CHARD, CHARD yields +5.3% / +17.2% / +13.3%. At BS8, MTP 5/1/6 gives 2219 and MTP 7/1/8 + CHARD gives 2216, a 0.1% difference that we regard as a tie.

Therefore, MTP + CHARD matches or beats the strongest MTP at every concurrency point (the gap is at most 0.3%, and it is 2.8% higher at BS16). The best configuration on 27B is MTP + CHARD, not plain CHARD-DFlash. This result directly shows that CHARD's consumption mechanism is not the bottleneck: when applied to a strong head, the system reaches that head's optimal throughput. CHARD is agnostic to the draft head, and DFlash's head-quality gap should not be attributed to CHARD.

Q3. Limited technical novelty; chunking lowers utilization

A3. The core of point 3 is that chunking gives speculative serving systems the ability to adapt to concurrency. Traditional block-diffusion serving binds the proposal width B, the verification granularity C, and request recovery into a single static configuration; CHARD decouples the three through chunked consumption and immediate redrafting, so the system can select the optimal configuration at each concurrency point and approach the theoretical bound for that concurrency.

The new data (per-concurrency B8/B4 table) show that N1–8 select B8 and N16–32 select B4; over the measured configuration space, the system's gap to the per-concurrency optimum is 0. A fixed B8 loses 3.9% at N32, and a fixed B4 loses 70.0% at N1. On DFlash, B8/C4 versus B4 gives +5.5% / +3.9% / +2.6% / +1.0% at BS1–8, +0.8% at BS16, and −1.0% at BS32; the system selects B4 at BS32 instead of forcing chunking. This is precisely what concurrency adaptivity means: CHARD does not require the same consumption mode at every point, but lets the system pick the optimum at each point.

The chunk-only ablation shows that decoupling chunking alone is insufficient. Chunk-only alone is negative (8B −12.7% / −3.0%, 27B −14.5% / −8.1%), with V/prop dropping from 8 to about 6 and calls/prop rising from 1 to about 2; T2 request-local recovery turns chunking into an adaptive capability, and only then does throughput return to and exceed one-shot B8. The mechanism of this paper is therefore chunking + recovery, not chunking alone.

This capability is independent of the draft head: MTP 7/1/8 + CHARD is +3.6% / +2.8% above the 3/1/4 baseline at BS8/BS16 and ties it at BS32, matching or exceeding the strongest MTP by up to 2.8%. This shows that CHARD provides system-level concurrency adaptivity rather than a DFlash-specific trick.

The residual gap to the hardware roofline bound comes from kernel efficiency, draft cost, and scheduling overhead; this is room for subsequent engineering optimization and does not affect the system's ability to select the optimal configuration at each concurrency point.

Q4. Writing clarity

A4. We acknowledge that the current version centers the narrative on two specific problems: wasted suffix computation, and TPOT differences caused by acceptance-rate heterogeneity. At the system level these two problems are subproblems of the same problem: existing block-diffusion serving binds the proposal width, the verification granularity, and request recovery into a single static configuration that cannot adapt to dynamic concurrency. Low concurrency needs long blocks and high concurrency needs short blocks; fixing either one loses at the other end.

Accordingly, the paper's narrative becomes: CHARD is a dynamically concurrency-adaptive serving system that decouples B, C, and recovery, selects a configuration according to concurrency, and keeps the system close to the per-concurrency optimum across the whole concurrency range; the original two problems become two manifestations of this system problem. Concrete changes: the abstract, introduction, contributions, and Section 7 are all rewritten around the "per-concurrency configuration envelope"; the two subproblems are retained as mechanism explanations rather than as the main thread. In addition, Table 1 gains configuration-family annotations, with B8/B4 as primary and B16 only as a same-width control; Sections 7.3/7.4 move the chunk-only negative result into the main text; and a new MTP + CHARD table is added.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.