- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CHARD modifies the proposal lifecycle in block-parallel speculative decoding to provide finer-grained target feedback and request-local recovery. Instead of verifying a full DFlash proposal in a single target call, it verifies the proposal in smaller chunks. Requests that pass a chunk retain the unverified proposal suffix, while requests that encounter a rejection can immediately redraft without waiting for other requests to finish consuming their proposals. The authors implement CHARD in SGLang and provide sufficient conditions under which this scheduling preserves the target distribution.
On Qwen3-8B and Qwen3.5-27B, CHARD reports up to 30.4% higher throughput and 27.4% lower request-level p99 TPOT than one-shot DFlash B16. An ablation comparing chunk-only verification with full CHARD shows that immediate redrafting provides additional throughput and latency gains beyond chunking alone.
Critical strengths
-
CHARD is implemented as an end-to-end serving mechanism rather than evaluated only through simulation or trace replay. The SGLang implementation maintains per-request proposal queues, retains valid suffixes across target calls, and allows replacement proposals to re-enter subsequent verification batches. It also handles the state and KV-cache management required for this asynchronous proposal lifecycle.
-
Table 2 cleanly separates the benefit of request-local recovery from chunked verification itself. On Qwen3-8B at BS32, throughput is 3,778 tokens/s with one-shot B8, 3,666 with chunk-only verification, and 3,971 with full CHARD. The two chunked variants have essentially identical evaluated positions and target calls per proposal, making the throughput difference a convincing isolation of the benefit of immediate redrafting rather than reduced verification work.
Critical weaknesses
-
The incremental benefit over adaptive verification is unclear. D-Cut and DSpark already address heterogeneous acceptance through adaptive verification. Table 2 isolates the benefit of CHARD's request-local recovery over fixed-width chunking, but the paper does not show whether this benefit remains when an existing adaptive verification policy is enabled. A comparison that adds CHARD-style suffix retention and recovery to an adaptive policy would clarify whether CHARD is complementary to, rather than substitutable for, prior methods.
-
The D-Cut comparison is not sufficiently controlled. D-Cut B8 with a 50% verification budget is evaluated in vLLM, whereas CHARD B8/C4 runs in SGLang. The D-Cut results are also averaged across concurrency settings while the text highlights a BS4 CHARD result. Thus, the reported numbers do not provide a matched head-to-head comparison. A same-runtime comparison using D-Cut's adaptive policy is needed to establish the relative benefit of CHARD.
-
The DSpark experiment does not compare against DSpark's verification scheduler. The experiment uses the DSpark draft head but compares one-shot verification with CHARD, rather than DSpark's confidence-scheduled verification. It therefore demonstrates compatibility with the DSpark drafter, not an advantage over DSpark's scheduling policy. Moreover, at BS32, CHARD improves low-acceptance median TPOT from 8.82 to 8.52 ms but worsens high-acceptance TPOT from 5.84 to 6.67 ms; this trade-off deserves further analysis.
-
The choice of chunk width C=4 is insufficiently justified. The paper states that a model-level sweep selected C=4, but does not report the sweep or its sensitivity to proposal width and concurrency. Figure 3 only estimates logical verification work and does not capture the runtime overhead of additional target calls. Since this trade-off is central to CHARD, end-to-end results across different C values are needed.
-
The serving workload is overly controlled. The evaluation uses fixed concurrency, decode-only execution, fixed 1,024-token outputs, and no early EOS. There is no arrival-driven workload with variable request lengths, despite CHARD's motivation being request-level recovery under heterogeneous progress. In addition, the reported p99 TPOT is a percentile of per-request average TPOT rather than individual inter-token gaps, so transient token-delivery stalls are not characterized.
Questions for authors
-
Can the authors compare D-Cut and CHARD under the same serving runtime, model, drafter, GPU, prompts, and concurrency settings? Please evaluate D-Cut with its native adaptive verification policy rather than only the reported 50% budget configuration, and report throughput and latency separately for each concurrency level.
-
Why does the DSpark-head experiment compare CHARD against one-shot verification rather than DSpark's confidence-scheduled verification? Does CHARD still provide additional benefit when DSpark's verification policy is enabled? A combined DSpark scheduling + CHARD recovery experiment would clarify their complementarity.
-
What were the results of the sweep used to select C=4? Please report throughput, TPOT, and target-call counts across several C values for both B8 and B16.
-
How does CHARD's scheduling and redrafting affect high-acceptance requests? In particular, what causes the increase in their median TPOT in the DSpark experiment, and is this a fundamental trade-off of request-local recovery or an artifact of the current scheduler?
-
How does CHARD perform under more realistic online serving conditions, including request arrivals, variable output lengths, and prefill/decode interference? It would also be useful to report a tail metric over individual inter-token gaps, separately from the p99 of request-average TPOT.
3 comments
Q3. Justification for C=4: C sweep at B8/B16
BS32 (macro-average over six workloads, tok/s / p99 TPOT ms):
| Proposal | C3 | C4 | C5 | C6 | C7 | C8 |
|---|---|---|---|---|---|---|
| B8 | 3789 / 8.31 | 3971 / 7.96 | 3773 / 8.43 | 3856 / 8.30 | 3741 / 8.56 | 3778 / 8.49 |
| B16 | 3700 / 8.54 | 3755 / 8.46 | 3718 / 8.61 | 3616 / 8.87 | 3563 / 8.95 | 3550 / 9.05 |
BS16:
| Proposal | C3 | C4 | C5 | C6 | C7 | C8 |
|---|---|---|---|---|---|---|
| B8 | 3221 / 4.97 | 3439 / 4.68 | 3395 / 4.80 | 3454 / 4.74 | 3272 / 5.05 | 3383 / 4.95 |
| B16 | 3096 / 5.21 | 3307 / 4.93 | 3299 / 4.99 | 3266 / 5.08 | 3131 / 5.27 | 3233 / 5.17 |
- BS32: both B8 and B16 peak at C4; B8 is 5.1% above C8, and B16 is 5.8% above C8.
- BS16: B16 peaks at C4; for B8, C4 and C6 are essentially tied (0.4%).
- C4 is not a globally fixed configuration; on the current hardware, low concurrency selects B8 (C=B) and high concurrency selects B8/C4.
Target calls per proposal:
- B8/C4 1.59, B8/C3 1.69, B8/C8 1.00;
- B16/C4 2.21, B16/C3 2.76, B16/C8 1.00.
C4 sits above C8 and below C3 in call count; it reaches the peak between "earlier recovery" and "additional calls."
Q4. TPOT regression for high-acceptance requests
This is not a harmful regression. What lowers SLO is the slow requests, not the high-acceptance requests that are already fast. The idea of this paper is to give up part of the fast requests' performance in exchange for the efficiency of the slow requests.
Under uniform C4, the median TPOT of high-acceptance requests rises from 5.84 to 6.67 ms; under the same mechanism:
- Qwen3-8B: TPOT of the lowest-acceptance quartile −26.0%, highest-acceptance quartile +1.9%;
- Qwen3.5-27B: lowest-acceptance quartile −27.7%, highest-acceptance quartile +5.9%.
The high-acceptance group pays about 2%–6% in TPOT, while the low-acceptance group gains about 26%–28%. SLO is determined by the latter, so the exchange is favorable. The 5.84→6.67 ms figure is the median on the DSpark workload; the bucketed mean over six workloads is +1.9%, consistent in direction.
Q5. More realistic online serving
1. Online conditions ultimately all act on instantaneous concurrency
Request arrivals, variable output lengths, and PD disaggregation all affect the decode side through one quantity: the number of active decode requests currently participating in model calls, N.
- High arrival rate → N increases; low arrival rate → N decreases.
- Long outputs keep requests in the decode pool longer → N increases at the same arrival rate; short outputs → N decreases.
- PD disaggregation changes the resource ratio between P and D, but N on the decode side is still determined by queueing in the D pool.
The system does not need to predict the arrival trace, nor does it need to know the output-length distribution; it reads N at runtime and looks up the optimal configuration for that N in the calibration table.
2. Calibration table: the optimal configuration for each N
DSpark-head calibration (432 points: six workloads, three runs, N1–32). The table below shows two representative configurations on Qwen3 8B on H20; changing hardware or model requires regenerating this calibration table:
| Concurrency | B8 optimized | B4 native | Best at this concurrency | vs fixed B8 | vs fixed B4 |
|---|---|---|---|---|---|
| 1 | 789 | 464 | B8 | 0 | +70.0% |
| 2 | 1428 | 889 | B8 | 0 | +60.6% |
| 4 | 2485 | 1652 | B8 | 0 | +50.4% |
| 8 | 3603 | 2915 | B8 | 0 | +23.6% |
| 16 | 4169 | 4220 | B4 | +1.2% | 0 |
| 32 | 4458 | 4632 | B4 | +3.9% | 0 |
- The optimum at N1–8 is B8; a fixed B4 loses 23.6%–70.0% at these points.
- The optimum at N16–32 is B4; a fixed B8 loses 1.2%–3.9% at these points.
- Per-concurrency selection takes the larger of the two at every N and is therefore never worse than either fixed configuration.
We note explicitly that this table is measured on the DSpark head, whereas the crossover figures quoted in the "System positioning" section are measured on DFlash (B4 higher by 4.3%/7.4% at BS16/32; B4 37.2% below B8 at N1; B16 below B8 by 19.3%/22.7%). The magnitudes differ because the two heads have different acceptance profiles and draft costs, which shifts where, and how sharply, the crossover occurs. The invariant across both heads is the crossover itself — B8 wins at low concurrency, B4 wins at high concurrency — and hence the need for per-concurrency selection. Our claims rest on that direction, not on any single magnitude.
3. Request arrivals
The arrival process determines the time series of N. The selector reads the current N at every scheduling step and looks up the configuration:
- Burst arrivals: N rises quickly and the selector switches to the high-concurrency configuration;
- Low arrival rate: N falls and the selector returns to B8, avoiding the loss of the high-concurrency configuration under low load.
The selector is reactive; it needs no arrival prediction, only the current value of N.
4. Variable output length
Output length affects N through residence time:
- Long outputs: requests stay in the decode pool longer, so N is larger at the same arrival rate, and the selector favors the high-N configuration;
- Short outputs: requests leave quickly, N is smaller, and the selector favors the low-N configuration.
The calibration table covers N1–32, and both directions fall inside it. The output-length distribution does not need separate modeling; its effect is already absorbed by N.
5. PD disaggregation
CHARD is a pure decode optimization and does not change the P rate.
- With the P rate unchanged and decode faster, the bottleneck shifts to P and the decode side may idle.
- This is a resource-ratio problem: adding P nodes or adjusting the P:D ratio converts the decode savings into system throughput.
- N on the decode side is still determined by queueing in the D pool, so the calibration table continues to apply; after scaling out P, the system can admit more requests, the operating range of N shifts upward, and the selector still picks configurations by N.
6. Individual inter-token gap reported separately from request-average TPOT
- Individual gap p99: the p99 over all single-token intervals;
- Request-average p99: first compute each request's mean commit gap, then take the p99 over all requests.
| Concurrency | CHARD gap p99 | CHARD req-avg p99 | B8 gap p99 | B8 req-avg p99 | gap diff | req-avg diff |
|---|---|---|---|---|---|---|
| 1 | 8.73 | 8.57 | 8.73 | 8.57 | 0% | 0% |
| 2 | 9.06 | 9.16 | 9.06 | 9.16 | 0% | 0% |
| 4 | 27.40 | 10.24 | 27.40 | 10.24 | 0% | 0% |
| 8 | 32.47 | 14.30 | 32.47 | 14.30 | 0% | 0% |
| 16 | 42.18 | 15.26 | 56.56 | 24.98 | −25.4% | −38.9% |
| 32 | 65.89 | 25.91 | 99.85 | 43.79 | −34.0% | −40.8% |
Response to Reviewer By1Z
We thank Reviewer By1Z for the detailed comments. We first state the system positioning of this paper, then respond point by point.
System positioning
Existing block-diffusion serving binds the proposal width B, the target verification granularity C, and the request recovery timing into a single static configuration. This configuration is not the same optimum under different concurrency. On the DFlash head, the crossover is as follows:
The same crossover appears on the DSpark head with different absolute magnitudes (see the calibration table in Q5.2): the optimum is B8 at N1–8 and B4 at N16–32, where a fixed B4 loses 23.6%–70.0% at low concurrency and a fixed B8 loses 1.2%–3.9% at high concurrency. What is invariant across the two heads is the direction of the crossover, not its size.
The system in this paper is not a single-point speedup on a fixed block, but an adaptive serving system: it decouples B, C, and the recovery timing, and selects the configuration according to concurrency. Concretely, for the hardware and workloads in this paper, the calibration result is B8 at low concurrency and B8/C4 at high concurrency; on different hardware or workloads the calibration table yields a different per-concurrency configuration, while the mechanism is unchanged.
The system is not tied to DFlash: on MTP 7/1/8, MTP+CHARD matches or beats the strongest MTP at every concurrency point.
Because absolute numbers depend on the draft head, we label the head wherever a result is head-specific: Q1 is measured on DFlash, and Q2 and Q5 on DSpark. Conclusions about the mechanism hold for both.
Q1. D-Cut vs. CHARD under identical runtime, model, drafter, GPU, prompts, and concurrency
We reproduced D-Cut in SGLang and compared D-Cut (auto) against CHARD at the B8 configuration, using the same DFlash head, a single H20, Qwen3 8B as the target, the same six workloads, and a sweep over concurrency:
Q2. Relation between the DSpark confidence scheduler and CHARD
DSpark and CHARD share the goal of reducing verification computation, but they target different things:
432-point comparison (same H20, same DSpark head, same prompts, six workloads, N1–32, three runs):
The two are currently not complementary: CHARD alone already exceeds DSpark+dynamic by 0.7%–5.3%. The reason is that they act on different stages of verification computation: DSpark reduces the tokens that enter verification, whereas CHARD changes how those tokens are consumed.
Because the stages differ, stacking the two is possible in principle: DSpark first prunes draft generation, then CHARD consumes the result in chunks. We leave this combination to future work.