- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DChord for speculative decoding under a fixed schema. PairDraft predicts field values and end-of-value markers in a compact representation and reconstructs the serialized draft through a schema-aware state machine. PairCut adjusts verification prefixes to exploit hardware latency plateaus. The evaluation shows promising efficiency gains. However, the main serving comparisons do not sufficiently isolate algorithmic improvements from task-specific training, and the stochastic verification procedure needs a more explicit justification.
Critical strengths
-
The problem is relevant and well motivated. The analysis of structural-token overhead identifies a concrete inefficiency in conventional speculative decoding for fixed-schema outputs. PairDraft addresses this inefficiency through a coherent design for compact drafting, variable-length value boundaries, and state recovery after verification.
-
The evaluation covers multiple target models and tasks, providing evidence that the efficiency gains are not confined to a single configuration. What's more, the DFlash-SFT ablation shows additional benefits beyond task fine-tuning in the single-request greedy setting.
Critical weaknesses
- The main efficiency claims require a task-adapted baseline comparison. DChord is trained on the evaluation tasks’ training splits, whereas the main speculative baselines use released checkpoints without reported equivalent task adaptation. The existing DFlash-SFT ablation is useful, but its single-request greedy setting does not establish whether the principal concurrent-serving gains persist under a comparable training setup. Please include DFlash-SFT in the main concurrent-serving evaluation, using the same task data, comparable training settings, and comparably tuned verification budgets. A consolidated table comparing released DFlash, DFlash-SFT, and DChord should clearly report throughput and relative improvements. The coverage should support the paper’s principal serving claims, including stochastic decoding where applicable.
The key question is whether DChord retains a meaningful advantage after controlling for task adaptation, rather than whether it outperforms the released checkpoints.
- The distribution-preservation claim is not sufficiently specified. Section 4.1 states that retaining standard target verification preserves the sampling distribution. However, compact predictions are expanded using predicted boundaries, and end-of-value markers are removed before verification. The paper does not explain the resulting proposal and verification procedure sufficiently clearly to check this claim. Please provide a concise, self-contained description of the implemented acceptance and rejection-correction procedure, specify the target distribution being preserved, and explain how draft expansion and any schema constraints satisfy the required correctness conditions.
Distribution preservation is an important claim. My concern is that its justification should be directly verifiable from the paper.
Questions for authors
-
After equivalent task adaptation and comparable verification-budget tuning, how much of DChord’s concurrent-serving advantage remains? Please provide the controlled comparison requested above.
-
What exact verification procedure is used for stochastic decoding, and why does it preserve the stated target distribution after draft expansion?
5 comments
W2: The distribution-preservation claim is not sufficiently specified. Section 4.1 states that retaining standard target verification preserves the sampling distribution. However, compact predictions are expanded using predicted boundaries, and end-of-value markers are removed before verification. The paper does not explain the resulting proposal and verification procedure sufficiently clearly to check this claim. Please provide a concise, self-contained description of the implemented acceptance and rejection-correction procedure, specify the target distribution being preserved, and explain how draft expansion and any schema constraints satisfy the required correctness conditions. Distribution preservation is an important claim. My concern is that its justification should be directly verifiable from the paper.
Q2: What exact verification procedure is used for stochastic decoding, and why does it preserve the stated target distribution after draft expansion?
Thank you for raising this point. Predicted end-of-value (EOV) markers affect draft construction without changing the target output distribution defined below. Section 4.1 did not clearly distinguish proposing a field boundary from committing it. PairDraft uses EOV markers to assemble the draft and removes them before verification. An EOV prediction does not change the target-side constraint state or make a value-closing token deterministic. The target still verifies the token that would close the current value. If that token is rejected, the remaining draft, including the next injected schema-key, is discarded. The PSM updates its state only from committed tokens.
In structured generation tasks, downstream applications rely on a predefined schema to parse the generated records, so constrained decoding enforces that schema[1]. DChord preserves the output distribution of the same target model under autoregressive decoding with constrained decoding.
In our implementation, the drafter deterministically selects schema-value tokens and EOV markers using argmax, regardless of the target decoding temperature. The PSM then deterministically expands these predictions into a complete draft. Thus, conditioned on the committed prefix and decoding state, the proposal distribution is a point mass at each proposed token, i.e., .
-
Schema-key positions. A predicted EOV marker only proposes a field transition; it does not make the boundary-closing token deterministic. Such a token (, a closing quotation mark for a variable-length string) is still subject to target verification and may be rejected. Once the field boundary has been confirmed, subsequent schema-determined key tokens are fixed by the target-side constraints and have target probability one. Thus, injecting schema-keys does not bypass target verification or alter the reference distribution.
-
Schema-value positions, including value-closing decisions. At T=0, the verifier accepts a proposed token only if it matches the constrained target's argmax, and otherwise emits the target's choice. At T>0, it applies standard speculative rejection sampling using the constrained target distribution and the deterministic proposal distribution . A proposed token is accepted with probability . Upon rejection, the remaining draft is discarded and a correction token is sampled from the residual target distribution. For the deterministic proposal , the residual distribution is for , and , whenever . Thus, the probability of emitting any token is , recovering the constrained target distribution exactly. If all proposed tokens are accepted, the bonus token is sampled from the constrained target distribution at the next position. The PSM advances only according to committed output tokens, not predicted EOV markers.
In the camera-ready version, we will explicitly define this reference distribution and provide a self-contained description of the stochastic acceptance and rejection-correction procedure, including the handling of injected structure and predicted boundaries.
[1] Liu M X, Liu F, Fiannaca A J, et al. " we need structured output": Towards user-centered constraints on large language model output[C]//Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 2024: 1-9.
Dear Reviewer Lape,
Thank you again for your constructive feedback. We would like to briefly summarize the additional evidence and clarifications provided in our responses.
-
W1/Q1 — Task-adapted concurrent-serving comparison. We added a controlled comparison of released DFlash(16), DFlash-SFT(16), and DChord(16) on Qwen3.5-4B at concurrency 16, using 512 requests per dataset under both greedy and stochastic decoding. DFlash-SFT uses the same task training data, training epochs, and optimizer settings as DChord. Under the same verification-width limit, DChord(16) retains average throughput gains of 11.20% at T=0 and 10.01% at T=1 over DFlash-SFT, calculated by averaging the per-dataset relative improvements across the four datasets.
-
W2/Q2 — Verification mechanism and distribution preservation. We clarified that EOV predictions cannot commit field boundaries without target verification and specified the acceptance, rejection-correction, and bonus-token rules, showing that DChord preserves the target model's output distribution under constrained decoding in both greedy and stochastic settings.
We would greatly appreciate your feedback on whether these additional results and clarifications address your concerns. Please let us know if any points would benefit from further explanation before the discussion closes.
Best regards, the authors of DChord
Thank you for suggesting this controlled concurrent-serving comparison.
DChord retains a throughput advantage over DFlash-SFT under concurrent serving. We fine-tuned the released DFlash checkpoint on the same task training data as DChord to obtain DFlash-SFT, using the same initialization checkpoint, number of training epochs, and optimizer settings reported for DChord in Appendix A.1 (Table 2). Both released DFlash and DFlash-SFT use the official DFlash(16) inference configuration, while DChord(16) is evaluated under the same 16-token target verification-width limit. We deliberately use this standardized configuration rather than performing method-specific verification-budget tuning, ensuring that the comparison isolates the effect of task adaptation and the proposed pair-level drafting mechanism under identical target verification budgets.
We evaluate released DFlash (16), DFlash-SFT (16), and DChord (16) on Qwen3.5-4B at concurrency 16 under both greedy (T=0) and stochastic (T=1) decoding, using 512 test requests per dataset at each temperature. DFlash-SFT(16) and DChord(16) use the same physical verification-width limit, allowing at most 15 draft tokens plus one correction/bonus query position per request.
Greedy decoding (T=0)
Stochastic decoding (T=1)
Throughput is measured in tokens/s, with speedup over AR at the same temperature shown in parentheses. Averaging the per-dataset relative throughput improvements, DChord achieves 11.20% and 10.01% higher throughput than DFlash-SFT at T=0 and T=1, respectively. These results show that DChord retains meaningful concurrent-serving gains after controlling for task-specific adaptation and target verification width, under both greedy and stochastic decoding.