acceptodds
Under review as a conference paper at ICLR 2027

DChord: Schema-Aware Speculative Decoding for Structured Generation

Abstract

Agents and embodied systems increasingly rely on large language models to produce structured generation for information extraction, tool invocation, and action commands. Recurring workloads are widely applied in practical scenarios, where requests from the same task share predefined field names, field order, and serialization, while the model predicts short, input-dependent field values. Recent work has explored speculative decoding to accelerate these structured generation workloads. However, these methods typically draft the complete serialized token sequence, allocating prediction positions to schema-determined structure and risking avoidable rejections on tokens that need not be predicted. We present DChord, a schema-aware speculative decoding framework with two components: PairDraft and PairCut. PairDraft concentrates prediction exclusively on field values and end markers, while a pair state machine supplies schema-determined structure to expand a compact value-only prediction into a complete draft. For concurrent serving, PairCut treats schema-determined structure as certain in its verification decisions, increasing acceptance length with negligible additional target latency. We evaluate DChord on Qwen3.5-4B, Qwen3.6-35B-A3B, and Gemma4-12B across four schema-determined tasks spanning multimodal and text-only inputs, comparing against autoregressive decoding (AR) and four recent speculative decoding methods. At concurrency 16, DChord achieves up to throughput speedup over AR and up to the throughput of the strongest competing speculative method.

Primary area:
generative models
Keywords:
Speculative Decoding, Schema-Aware, Structured Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LapeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DChord for speculative decoding under a fixed schema. PairDraft predicts field values and end-of-value markers in a compact representation and reconstructs the serialized draft through a schema-aware state machine. PairCut adjusts verification prefixes to exploit hardware latency plateaus. The evaluation shows promising efficiency gains. However, the main serving comparisons do not sufficiently isolate algorithmic improvements from task-specific training, and the stochastic verification procedure needs a more explicit justification.

Critical strengths
  1. The problem is relevant and well motivated. The analysis of structural-token overhead identifies a concrete inefficiency in conventional speculative decoding for fixed-schema outputs. PairDraft addresses this inefficiency through a coherent design for compact drafting, variable-length value boundaries, and state recovery after verification.

  2. The evaluation covers multiple target models and tasks, providing evidence that the efficiency gains are not confined to a single configuration. What's more, the DFlash-SFT ablation shows additional benefits beyond task fine-tuning in the single-request greedy setting.

Critical weaknesses
  1. The main efficiency claims require a task-adapted baseline comparison. DChord is trained on the evaluation tasks’ training splits, whereas the main speculative baselines use released checkpoints without reported equivalent task adaptation. The existing DFlash-SFT ablation is useful, but its single-request greedy setting does not establish whether the principal concurrent-serving gains persist under a comparable training setup. Please include DFlash-SFT in the main concurrent-serving evaluation, using the same task data, comparable training settings, and comparably tuned verification budgets. A consolidated table comparing released DFlash, DFlash-SFT, and DChord should clearly report throughput and relative improvements. The coverage should support the paper’s principal serving claims, including stochastic decoding where applicable.

The key question is whether DChord retains a meaningful advantage after controlling for task adaptation, rather than whether it outperforms the released checkpoints.

  1. The distribution-preservation claim is not sufficiently specified. Section 4.1 states that retaining standard target verification preserves the sampling distribution. However, compact predictions are expanded using predicted boundaries, and end-of-value markers are removed before verification. The paper does not explain the resulting proposal and verification procedure sufficiently clearly to check this claim. Please provide a concise, self-contained description of the implemented acceptance and rejection-correction procedure, specify the target distribution being preserved, and explain how draft expansion and any schema constraints satisfy the required correctness conditions.

Distribution preservation is an important claim. My concern is that its justification should be directly verifiable from the paper.

Questions for authors
  1. After equivalent task adaptation and comparable verification-budget tuning, how much of DChord’s concurrent-serving advantage remains? Please provide the controlled comparison requested above.

  2. What exact verification procedure is used for stochastic decoding, and why does it preserve the stated target distribution after draft expansion?

5 comments

W1: The main efficiency claims require a task-adapted baseline comparison. DChord is trained on the evaluation tasks' training splits, whereas the main speculative baselines use released checkpoints without reported equivalent task adaptation. The existing DFlash-SFT ablation is useful, but its single-request greedy setting does not establish whether the principal concurrent-serving gains persist under a comparable training setup. Please include DFlash-SFT in the main concurrent-serving evaluation, using the same task data, comparable training settings, and comparably tuned verification budgets. A consolidated table comparing released DFlash, DFlash-SFT, and DChord should clearly report throughput and relative improvements. The coverage should support the paper's principal serving claims, including stochastic decoding where applicable. The key question is whether DChord retains a meaningful advantage after controlling for task adaptation, rather than whether it outperforms the released checkpoints.

Q1: After equivalent task adaptation and comparable verification-budget tuning, how much of DChord’s concurrent-serving advantage remains? Please provide the controlled comparison requested above.

Thank you for suggesting this controlled concurrent-serving comparison.

DChord retains a throughput advantage over DFlash-SFT under concurrent serving. We fine-tuned the released DFlash checkpoint on the same task training data as DChord to obtain DFlash-SFT, using the same initialization checkpoint, number of training epochs, and optimizer settings reported for DChord in Appendix A.1 (Table 2). Both released DFlash and DFlash-SFT use the official DFlash(16) inference configuration, while DChord(16) is evaluated under the same 16-token target verification-width limit. We deliberately use this standardized configuration rather than performing method-specific verification-budget tuning, ensuring that the comparison isolates the effect of task adaptation and the proposed pair-level drafting mechanism under identical target verification budgets.

We evaluate released DFlash (16), DFlash-SFT (16), and DChord (16) on Qwen3.5-4B at concurrency 16 under both greedy (T=0T=0) and stochastic (T=1T=1) decoding, using 512 test requests per dataset at each temperature. DFlash-SFT(16) and DChord(16) use the same physical verification-width limit, allowing at most 15 draft tokens plus one correction/bonus query position per request.

Greedy decoding (T=0T=0)

MethodCelebA τ\tauCelebA throughputKAGL τ\tauKAGL throughputCUB τ\tauCUB throughputASAP τ\tauASAP throughput
AR-881.21 (1.00x)-535.33 (1.00x)-817.33 (1.00x)-784.59 (1.00x)
DFlash (16)8.442109.06 (2.39x)6.40682.23 (1.27x)8.171785.90 (2.19x)11.581810.62 (2.31x)
DFlash-SFT (16)14.243029.80 (3.44x)7.79689.50 (1.29x)12.442156.70 (2.64x)12.051906.00 (2.43x)
DChord (16)15.293199.89 (3.63x)11.08820.56 (1.53x)14.962439.49 (2.98x)14.302040.92 (2.60x)

Stochastic decoding (T=1T=1)

MethodCelebA τ\tauCelebA throughputKAGL τ\tauKAGL throughputCUB τ\tauCUB throughputASAP τ\tauASAP throughput
AR-889.99 (1.00x)-518.36 (1.00x)-816.61 (1.00x)-786.39 (1.00x)
DFlash (16)8.402063.37 (2.32x)6.28678.09 (1.31x)5.981385.90 (1.70x)10.971770.18 (2.25x)
DFlash-SFT (16)13.722909.90 (3.27x)7.33679.40 (1.31x)9.041700.30 (2.08x)11.251792.66 (2.28x)
DChord (16)14.873127.80 (3.51x)8.57782.43 (1.51x)10.921886.20 (2.31x)12.881908.27 (2.43x)

Throughput is measured in tokens/s, with speedup over AR at the same temperature shown in parentheses. Averaging the per-dataset relative throughput improvements, DChord achieves 11.20% and 10.01% higher throughput than DFlash-SFT at T=0 and T=1, respectively. These results show that DChord retains meaningful concurrent-serving gains after controlling for task-specific adaptation and target verification width, under both greedy and stochastic decoding.

W2: The distribution-preservation claim is not sufficiently specified. Section 4.1 states that retaining standard target verification preserves the sampling distribution. However, compact predictions are expanded using predicted boundaries, and end-of-value markers are removed before verification. The paper does not explain the resulting proposal and verification procedure sufficiently clearly to check this claim. Please provide a concise, self-contained description of the implemented acceptance and rejection-correction procedure, specify the target distribution being preserved, and explain how draft expansion and any schema constraints satisfy the required correctness conditions. Distribution preservation is an important claim. My concern is that its justification should be directly verifiable from the paper.

Q2: What exact verification procedure is used for stochastic decoding, and why does it preserve the stated target distribution after draft expansion?

Thank you for raising this point. Predicted end-of-value (EOV) markers affect draft construction without changing the target output distribution defined below. Section 4.1 did not clearly distinguish proposing a field boundary from committing it. PairDraft uses EOV markers to assemble the draft and removes them before verification. An EOV prediction does not change the target-side constraint state or make a value-closing token deterministic. The target still verifies the token that would close the current value. If that token is rejected, the remaining draft, including the next injected schema-key, is discarded. The PSM updates its state only from committed tokens.

In structured generation tasks, downstream applications rely on a predefined schema to parse the generated records, so constrained decoding enforces that schema[1]. DChord preserves the output distribution of the same target model under autoregressive decoding with constrained decoding.

In our implementation, the drafter deterministically selects schema-value tokens and EOV markers using argmax, regardless of the target decoding temperature. The PSM then deterministically expands these predictions into a complete draft. Thus, conditioned on the committed prefix and decoding state, the proposal distribution is a point mass at each proposed token, i.e., q=δxq=\delta_x.

  • Schema-key positions. A predicted EOV marker only proposes a field transition; it does not make the boundary-closing token deterministic. Such a token (e.g.e.g., a closing quotation mark for a variable-length string) is still subject to target verification and may be rejected. Once the field boundary has been confirmed, subsequent schema-determined key tokens are fixed by the target-side constraints and have target probability one. Thus, injecting schema-keys does not bypass target verification or alter the reference distribution.

  • Schema-value positions, including value-closing decisions. At T=0, the verifier accepts a proposed token only if it matches the constrained target's argmax, and otherwise emits the target's choice. At T>0, it applies standard speculative rejection sampling using the constrained target distribution pp and the deterministic proposal distribution q=δxq=\delta_x. A proposed token xx is accepted with probability p(x)p(x). Upon rejection, the remaining draft is discarded and a correction token is sampled from the residual target distribution. For the deterministic proposal q=δxq=\delta_x, the residual distribution is r(y)=p(y)/(1−p(x))r(y)=p(y)/(1-p(x)) for y≠xy\neq x, and r(x)=0r(x)=0, whenever p(x)<1p(x)<1. Thus, the probability of emitting any token yy is p(x)1[y=x]+(1−p(x))r(y)=p(y)p(x)\mathbf{1}[y=x]+(1-p(x))r(y)=p(y), recovering the constrained target distribution exactly. If all proposed tokens are accepted, the bonus token is sampled from the constrained target distribution at the next position. The PSM advances only according to committed output tokens, not predicted EOV markers.

In the camera-ready version, we will explicitly define this reference distribution and provide a self-contained description of the stochastic acceptance and rejection-correction procedure, including the handling of injected structure and predicted boundaries.

[1] Liu M X, Liu F, Fiannaca A J, et al. " we need structured output": Towards user-centered constraints on large language model output[C]//Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 2024: 1-9.

Dear Reviewer Lape,

Thank you again for your constructive feedback. We would like to briefly summarize the additional evidence and clarifications provided in our responses.

  • W1/Q1 — Task-adapted concurrent-serving comparison. We added a controlled comparison of released DFlash(16), DFlash-SFT(16), and DChord(16) on Qwen3.5-4B at concurrency 16, using 512 requests per dataset under both greedy and stochastic decoding. DFlash-SFT uses the same task training data, training epochs, and optimizer settings as DChord. Under the same verification-width limit, DChord(16) retains average throughput gains of 11.20% at T=0 and 10.01% at T=1 over DFlash-SFT, calculated by averaging the per-dataset relative improvements across the four datasets.

  • W2/Q2 — Verification mechanism and distribution preservation. We clarified that EOV predictions cannot commit field boundaries without target verification and specified the acceptance, rejection-correction, and bonus-token rules, showing that DChord preserves the target model's output distribution under constrained decoding in both greedy and stochastic settings.

We would greatly appreciate your feedback on whether these additional results and clarifications address your concerns. Please let us know if any points would benefit from further explanation before the discussion closes.

Best regards, the authors of DChord

Reviewer_iN4WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DChord, a schema-aware speculative decoding framework for structured generation. It consists of two components: PairDraft, which predicts only input-dependent field values and their boundaries while injecting schema-determined tokens, and PairCut, which exploits deterministic structure and hardware-aware verification costs to select verification lengths. Experiments across three target models and four structured generation tasks demonstrate up to 3.63× throughput over autoregressive decoding and 1.63× over competing speculative decoding methods.

Critical strengths
  1. Simple and effective approach: The paper identifies an important inefficiency in conventional speculative decoding for structured outputs. Predicting only field values and injecting deterministic schema tokens is a well-motivated approach to improving draft efficiency and acceptance length.

  2. Convincing empirical improvements: The authors demonstrate substantial improvements across dense and MoE models, multiple tasks, and sampling temperatures. Importantly, comparisons against task-fine-tuned DFlash and schema-aware autoregressive decoding help isolate the benefits of the proposed method.

  3. Hardware-aware verification: PairCut provides a practical mechanism for exploiting additional verification capacity at negligible incremental latency, with improvements over confidence-based verification pruning, particularly at higher concurrency.

Critical weaknesses
  1. Limited evaluation of generalizability: The evaluation focuses on recurring workloads with fixed schemas and short field values. It is unclear how effectively the method extends to structured generation with variable schemas, optional fields, or longer values, limiting the demonstrated scope of its applicability.

  2. Relatively modest contribution of PairCut: While PairDraft provides substantial improvements, PairCut improves throughput over confidence-based pruning by only 2.24% at concurrency 16, with larger gains emerging at higher concurrency. This makes its practical significance less clear under moderate serving loads.

Questions for authors
  1. How well does PairDraft generalize to unseen schemas or schemas with variable field ordering and longer values? Would these scenarios require additional fine-tuning?

  2. Could the authors compare against a simpler schema-aware DFlash baseline that explicitly injects deterministic structure while retaining conventional token-level drafting? This would help isolate the contribution of the compressed pair-level representation.

  3. Under which concurrency and verification-cost regimes does PairCut provide meaningful benefits over confidence-based pruning? Can the authors provide more insight into its relatively modest gains at concurrency 16?

6 comments

W1: Limited evaluation of generalizability: The evaluation focuses on recurring workloads with fixed schemas and short field values. It is unclear how effectively the method extends to structured generation with variable schemas, optional fields, or longer values, limiting the demonstrated scope of its applicability.

Q1: How well does PairDraft generalize to unseen schemas or schemas with variable field ordering and longer values? Would these scenarios require additional fine-tuning?

Thank you for recognizing DChord's key insight that predicting only field values and injecting deterministic schema tokens can improve drafting efficiency and increase acceptance length.

Regarding generalizability, we investigate four scenarios: variable field ordering, optional fields, previously unseen schemas, and longer values. For each scenario, we provide the corresponding mechanistic analysis or experiments. The experiments use Qwen3.5-4B under single-request greedy decoding and report acceptance length.

1. Variable Order

In practice, schema-based workloads often use non-streaming interactions [1]. Downstream applications parse the complete record before acting on it and can reorder fields after parsing. A predefined field order is therefore sufficient for these workloads.

However, DChord has the capability to accommodate changes in field order without additional fine-tuning. The task instructions and one-shot example specify the field order, and the PSM is registered with that same order. Both can be updated when the order changes. To evaluate this capability, we randomly select and shuffle 50% of the schema pairs in CelebA and CUB, update the instructions and one-shot example to reflect the new order, and re-register the PSM while keeping the drafter weights unchanged.

Field orderCUB: DFlash(16)CUB: DChordCelebA: DFlash(16)CelebA: DChord
Original8.05623.1418.44224.999
Reordered7.19720.0568.00523.346

Under the reordered schemas, without additional fine-tuning, DChord still achieves a higher average acceptance length than DFlash. Acceptance length changes for both methods, consistent with the fact that reordered field order changes the context for subsequent predictions [2,3,4] and can therefore affect agreement between the drafter and the target model.

2. Optional Fields

We distinguish pre-specified optional fields from dynamically determined optional fields. DChord supports the former, where the field subset is fixed before decoding. The latter, where field inclusion is decided during generation, falls outside our scope of recurring tasks with a predefined output structure.

DChord retains its acceptance length advantage when the user selects a subset of fields before generation. This is because the one-shot example in the prompt conveys the requested schema to the drafter, and the PSM uses the corresponding template to assemble drafts. On CelebA and CUB, we randomly retain 50% of the fields while preserving their relative order. We update the prompt and re-register the PSM for the reduced schema without fine-tuning the drafter.

Field configurationCUB: DFlash (16)CUB: DChordCelebA: DFlash (16)CelebA: DChord
Full schema8.05623.1418.44224.999
Retain 50% of fields7.21919.0988.24523.714

With 50% of the fields randomly retained, DChord achieves 2.65× the acceptance length of DFlash on CUB and 2.88× on CelebA. These results show that the same drafter can be reused without additional training when the field subset is specified before generation. As with field reordering, removing fields changes the context for subsequent predictions, which may affect agreement between the drafter and the target and explain why acceptance length varies for both methods.

[1] Liu M X, Liu F, Fiannaca A J, et al. " we need structured output": Towards user-centered constraints on large language model output[C]//Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 2024: 1-9.

[2] Liu H, Yang M, Adomavicius G. Robustness is important: Limitations of LLMs for predictions on tabular data[J]. PNAS nexus, 2026, 5(6): pgag197.

[3] Le Y. Schema Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding[J]. arXiv preprint arXiv:2604.14862, 2026.

[4] Kumar V, Mandora B. Structure Tax: How Structured Output affects LLMs Performance[C]//Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 4620-4644.

Continue with W1/Q1:

3. Unseen Schema

DChord can adapt to unseen schemas through lightweight task-specific fine-tuning. We demonstrate this capability on Memotion [5], a multimodal sentiment analysis task whose output contains five fields with short free-text values ranging from 1 to 5 tokens (2.76 on average). We register the new schema in DChord's PSM and compare DFlash and DChord before and after SFT. Both methods use the same SFT configuration following Appendix A.1. We evaluate both methods on 128 held-out examples. The acceptance lengths are as follows:

DFlash(16)DChordDFlash-SFT(16)DChord-SFT(16)DChord-SFT
5.9114.2819.76511.76616.795

Before SFT, DChord has a lower acceptance length than the released DFlash, indicating that registering a new schema alone is insufficient to maintain drafting efficiency on a completely new task.

Both methods improve after SFT, and DChord achieves a higher acceptance length than DFlash after fine-tuning. DChord-SFT achieves an average acceptance length of 16.795, compared with 9.765 for DFlash-SFT (16). Even under the same limit of 15 draft tokens per round, DChord-SFT (16) reaches 11.766, about 20.5% higher than DFlash-SFT (16).

For Memotion, a new and unseen task, DChord can establish an acceptance length advantage on this unseen task through low-cost SFT. Specifically, SFT is performed on two NVIDIA A800-SXM4-80GB GPUs with 4k training samples, and DChord's three-epoch training stage takes approximately 45 minutes.

4. Longer Values

DChord can draft a single schema-value across multiple rounds. When the current value fills all 15 draft positions without an end-of-value marker, the Pair State Machine (PSM) submits those value tokens for verification without inserting the next schema-key. If the committed output still ends within that value, the next draft starts from the latest committed position.

We evaluate DChord on HotpotQA [6], a multi-hop question answering task in which the model reads a question and ten Wikipedia passages and returns five key–value pairs. The full evaluation set contains 128 examples, with a mean schema-value length of 6.07 tokens. Schema-values account for 54% of output tokens. The long-value subset consists of the 15 examples with at least one value longer than 15 tokens. On this subset, the mean schema-value length is 9.28 tokens, and schema-values account for 65% of output tokens.

We fine-tune DChord and released DFlash on the same 4,000 HotpotQA training examples, following Appendix A.1 for the remaining training settings. We evaluate both on Qwen3.5-4B at concurrency 1 under greedy decoding with identical output-format constraints. Both methods use 15 draft prediction positions. DChord-SFT verifies its expanded draft, which can exceed 15 tokens after schema-key injection.

MethodFull test set (128 examples)Long-value subset (15 examples)
DFlash-SFT(16)10.16610.859
DChord-SFT12.25512.500

DChord-SFT achieves a higher average acceptance length than DFlash-SFT (16) on both sets, committing 2.089 more tokens per round on the full set and 1.641 more on the long-value subset. The advantage persists even when longer values leave fewer opportunities to advance through injected schema-keys.

[5] Jin Y, Choi M, Verma G, et al. Mm-soc: Benchmarking multimodal large language models in social media platforms[C]//Findings of the Association for Computational Linguistics: ACL 2024. 2024: 6192-6210.

[6] Yang Z, Qi P, Zhang S, et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering[C]//Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 2369-2380.

Q2: Could the authors compare against a simpler schema-aware DFlash baseline that explicitly injects deterministic structure while retaining conventional token-level drafting? This would help isolate the contribution of the compressed pair-level representation.

Thank you for suggesting this comparison. Released DFlash does not support value-only drafting across multiple schema fields because it lacks DChord's explicit schema-state tracking and value-boundary representation. In DChord, the DSE provides the current decoding position, the drafter predicts values and their end markers, and the PSM uses those markers to insert the corresponding keys.

To address the request for a simpler schema-aware DFlash baseline, we implemented two schema-injection baselines while keeping the released drafter unchanged: Pair-Schema-Aware DFlash, which uses injected keys and drafts one value per round, and Schema-Aware DFlash, which drafts both keys and values and then replaces the generated keys with pre-known schema-keys.

Pair-Schema-Aware DFlash uses only the draft proposals for the current value. Once the target confirms the value boundary, the state machine injects the next schema-key and starts another drafting round. As detailed in our response to Reviewer gAWc's Q2, its average acceptance length decreases from 8.483 to 6.469, because proposals beyond the current value boundary are discarded and drafting is interrupted between fields.

Schema-Aware DFlash uses the pre-known schema-keys to correct drafted schema-keys while preserving proposals across multiple fields within the same round. After DFlash proposes 15 tokens, a schema state machine identifies key positions and replaces them with the pre-known schema-keys, retaining value proposals at their original positions. This procedure relies on the draft proposal and schema information, without using ground-truth field boundaries or future target-model outputs. We also evaluate DFlash-SFT(16) and apply the same injection procedure to its checkpoint to obtain Schema-Aware DFlash-SFT(16). Both schema-aware variants limit the expanded candidate to 15 tokens. We evaluate Qwen3.5-4B under the single-request and greedy decoding setup. All methods use 15 draft prediction positions. We additionally report DChord(16), which limits target verification to the same 16-token width as the DFlash(16) baselines.

DatasetDFlash(16)DFlash-SFT(16)Pair-Schema-Aware DFlash(16)Schema-Aware DFlash(16)Schema-Aware DFlash-SFT(16)DChord(16)DChord
CelebA8.44214.2138.10012.98314.88615.25624.999
KAGL6.1018.0073.8067.9259.05611.11215.613
CUB8.05612.3355.3879.48812.66714.88023.141
ASAP11.33411.9388.58312.53912.97614.17522.378
AVG8.48311.6236.46910.73412.39613.85621.533

Schema-Aware DFlash(16) achieves 1.27× the average acceptance length of released DFlash(16). With the same injection procedure and verification capacity, the fine-tuned checkpoint raises the average from 10.734 to 12.396 tokens per round (+15.5%). Schema-Aware DFlash-SFT(16) also achieves 6.7% higher acceptance length than DFlash-SFT(16). Under the same 16-token target verification limit, DChord(16) achieves 13.856 tokens per round, improving on Schema-Aware DFlash(16) by 29.1% and on Schema-Aware DFlash-SFT(16) by 11.8%. DChord(16) leads the latter on all four datasets. With its full expanded verification width, DChord reaches 21.533 tokens per round, or 1.74× the acceptance length of Schema-Aware DFlash-SFT(16).

These results highlight the advantage of compressed pair-level drafting even when DFlash benefits from both task fine-tuning and schema-key injection. In Schema-Aware DFlash, deterministic tokens still consume draft prediction positions even when subsequently corrected. PairDraft concentrates its prediction budget on schema-values and their boundaries, allowing the same number of draft prediction positions to cover more fields. The expanded proposals can then be verified with a larger target verification budget for further gains.

Reviewer_gAWcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies schema keys that are fixed and structural, such as answer templates, so they can be reused across different inputs in recurring structured generation workloads. Other than schema keys, there are schema values that are dynamic and depend on inputs, such as answers to fill in the answer template. This paper proposes that the drafter should focus on predicting schema values to better utilize drafter capacity. This paper proposes PairDraft where the drafter only predicts schema values, which later will be aggregated with predefined schema keys to form the completed sequence for target verification. PairDraft predicts multiple schema-values in parallel which are divided by different schema-keys. PairCut is also proposed to dynamically determine the draft generated token length.

Critical strengths
  1. This paper identifies schema keys can be skipped from predicting by the drafter and only schema values need to be predicted.
  2. Substantial accept length and speedup improvements are achieved across different datasets and models.
Critical weaknesses
  1. The DSE only encodes the one last schema key-value pair's information as input for drafting based on line 226-232. But PairDraft will predict next several schema-values that across different schema-keys based on Figure 2b. How can the draft predict the schema values without knowing what the corresponding schema keys are? This need to be clarified
  2. In this paper's setting, the schema-keys are pre-known, so DChord drafter does not predict them. In the reported accept length of DChord, the pre-known schema-keys are counted, while other methods still need to predict such pre-known schema-keys, making the comparison unfair. Since the schema-keys are pre-known, why other methods need to predict them?
  3. The description of PairCut is very difficult to follow and understand, there are too many symbol definitions. Equations are heuristic and lack theoretical prove.
Questions for authors
  1. The DSE only encodes the one last schema key-value pair's information as input for drafting based on line 226-232. But PairDraft will predict next several schema-values that across different schema-keys based on Figure 2b. How can the draft predict the schema values without knowing what the corresponding schema keys are? Does it indicate the schema values are less relevant to schema-keys?
  2. What is the accept length when Dflash is given pre-known schema-keys and only predicts the schema-values?
  3. What are the mean, P25, P75 lengths of schema-values in these datasets?
8 comments

W1: The DSE only encodes the one last schema key-value pair's information as input for drafting based on line 226-232. But PairDraft will predict next several schema-values that across different schema-keys based on Figure 2b. How can the draft predict the schema values without knowing what the corresponding schema keys are? This need to be clarified

Q1: The DSE only encodes the one last schema key-value pair's information as input for drafting based on line 226-232. But PairDraft will predict next several schema-values that across different schema-keys based on Figure 2b. How can the draft predict the schema values without knowing what the corresponding schema keys are? Does it indicate the schema values are less relevant to schema-keys?

Thank you for pointing out this passage that requires clarification.

Importantly, the Decoding State Encoder (DSE) is not intended to encode the identities of all future fields. It encodes the current decoding cursor, while the target context features provide the semantics and ordering of the complete schema through the prompt and committed prefix. For example, starting from the current pair index jj, the kk-th subsequent value segment is associated with field j+kj+k in the predefined schema order. Thus, the DSE only needs to identify the starting pair, rather than explicitly encode all future fields. As described in Lines 240–244, the prompt specifies the field names and their order through task instructions and a one-shot example. Thus, the drafter can access information about subsequent fields through the target context features, even before generation reaches those fields.

During training, the drafter learns an ordered compressed representation of the form vjv_j, ⟨END_VALUE⟩, vj+1v_{j+1}, ⟨END_VALUE⟩, …\ldots. Each predicted END_VALUE (end-of-value, EOV) marker separates consecutive field values, allowing the subsequent value segment to be associated with the next field in the predefined schema order.

Therefore, schema-values are not independent of schema-keys. Their dependence on field semantics is modeled through the target context features, while the DSE identifies the starting state and the predicted end markers determine field transitions. The Pair State Machine (PSM) then uses these boundaries to reconstruct the complete serialized draft.

For example, suppose the prompt specifies the field order color → size → material and includes a one-shot example such as {"color":"red","size":"medium","material":"cotton"}. The drafter receives the field names and their order through the target context features computed from this prompt. Now suppose the committed output ends at {"color":"dark. If dark occupies one token, the DSE indicates that one token of the color value has been committed and that the anchor is a schema-value token. With the field order from the prompt and the current position from the DSE, PairDraft can continue the color value (e.g.e.g., following with blue) and predict values for size and material along with their end-of-value markers in the same drafting round.

We will also revise the relevant wording in the paper to make this explanation clearer.

W2: In this paper's setting, the schema-keys are pre-known, so DChord drafter does not predict them. In the reported accept length of DChord, the pre-known schema-keys are counted, while other methods still need to predict such pre-known schema-keys, making the comparison unfair. Since the schema-keys are pre-known, why other methods need to predict them?

Q2: What is the accept length when Dflash is given pre-known schema-keys and only predicts the schema-values?

Thank you for raising this point. We use the standard speculative decoding definition of acceptance length, as in EAGLE-3 [1]: the average number of new tokens committed to the final output per drafting and verification round. We apply this definition uniformly to all methods. In DChord, both predicted tokens and injected schema-keys count only if they are committed by the verification procedure. This provides a consistent measure of final-output progress per round.

To further address the fairness concern, we evaluate whether DFlash can similarly benefit from pre-known schema-keys. The key challenge is that knowing the next schema-key does not determine where the current schema-value ends. Released DFlash uses token-level drafting and has no mechanism for marking value boundaries in a compressed draft. In contrast, DChord jointly predicts schema-values and EOV markers, while its DSE identifies the current decoding position. The PSM then uses the predicted boundaries to insert schema-keys, allowing DChord to draft across multiple fields without spending prediction slots on known structure.

To satisfy the requirements in Q2 "Dflash is given pre-known schema-keys and only predicts the schema-values", we implemented Pair-Schema-Aware DFlash. Specifically, each round drafts only the current schema-value for the current schema field. Once the target confirms its boundary, the state machine injects the next schema-key and starts a new drafting round. The acceptance lengths for Qwen3.5-4B across the four datasets are shown as follows.

DatasetDFlash(16)Pair-Schema-Aware DFlash(16)DChord
CelebA8.4428.10024.999
KAGL6.1013.80615.613
CUB8.0565.38723.141
ASAP11.3348.58322.378
AVG8.4836.46921.533

Pair-Schema-Aware DFlash(16) reaches an average acceptance length of 6.469, compared with 8.483 for released DFlash(16). This is because released DFlash can use proposals spanning multiple fields in one round, whereas pair-by-pair injection discards proposals beyond the current value. DChord injects the known keys while retaining the ability to draft across fields, reaching an average acceptance length of 21.533.

[1] Li Y, Wei F, Zhang C, et al. Eagle-3: Scaling up inference acceleration of large language models via training-time test[J]. Advances in Neural Information Processing Systems, 2026, 38: 136737-136756.

Continue with W2Q2:

We further evaluate a Schema-Aware DFlash baseline that explicitly injects deterministic structure while retaining conventional token-level drafting, as mentioned in Reviewer iN4W's Q2. Specifically, it uses the pre-known schema-keys to correct drafted schema-keys while preserving proposals across multiple fields within the same round. After DFlash proposes 15 tokens, a schema state machine identifies key positions and replaces them with the pre-known schema-keys, retaining value proposals at their original positions. This procedure relies on the draft proposal and schema information, without using ground-truth field boundaries or future target-model outputs. We also include DFlash-SFT(16) and apply the same injection procedure to its checkpoint to obtain Schema-Aware DFlash-SFT(16). Both schema-aware variants cap the expanded candidate at 15 tokens. The results are shown as follows.

DatasetDFlash(16)DFlash-SFT(16)Pair-Schema-Aware DFlash(16)Schema-Aware DFlash(16)Schema-Aware DFlash-SFT(16)DChord(16)DChord
CelebA8.44214.2138.10012.98314.88615.25624.999
KAGL6.1018.0073.8067.9259.05611.11215.613
CUB8.05612.3355.3879.48812.66714.88023.141
ASAP11.33411.9388.58312.53912.97614.17522.378
AVG8.48311.6236.46910.73412.39613.85621.533

Schema-Aware DFlash(16) achieves 1.27× the average acceptance length of released DFlash(16). With the same injection procedure and verification capacity, using the fine-tuned checkpoint increases acceptance length from 10.734 to 12.396 tokens per round (+15.5%). This is 6.7% higher than DFlash-SFT(16). At the same verification capacity, DChord(16) reaches 13.856, outperforming Schema-Aware DFlash(16) by 29.1% and Schema-Aware DFlash-SFT(16) by 11.8%. DChord(16) leads the latter on all four datasets. With its full expanded verification width, DChord reaches 21.533, or 1.74× the acceptance length of Schema-Aware DFlash-SFT(16).

These results highlight the advantage of compressed pair-level drafting even when the DFlash baseline benefits from both task fine-tuning and schema-key injection. In Schema-Aware DFlash, deterministic tokens still consume draft prediction positions even when subsequently corrected. PairDraft concentrates its prediction budget on schema-values and their boundaries, allowing the same number of draft prediction positions to cover more fields. The expanded proposals can then be verified with a larger target verification budget for further gains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers gAWc, iN4W, and Lape as R1, R2, and R3, respectively.

We sincerely thank all three reviewers for their careful reading and constructive feedback. R1, R2, and R3 all recognized two strengths: (1) the rationale behind DChord, namely that predicting deterministic structure incurs avoidable drafting overhead, and (2) DChord's gains across datasets, dense and MoE models, and sampling temperatures. R2 and R3 also recognized the value of the comparison with DFlash-SFT. R2 highlighted DChord's simplicity and effectiveness and noted the practical benefits of PairCut at higher concurrency. R3 praised DChord's design, which combines a compressed pair-level representation with boundary handling for variable-length values and state recovery after verification.


Core Contributions

  • DChord: schema-aware speculative decoding framework. DChord consists of PairDraft and PairCut to achieve efficient speculative decoding for structured generation. PairDraft addresses the challenge of drafting values without predicting schema-determined structure. PairCut further controls the increasing verification cost under concurrency.

  • PairDraft: compressed pair-level drafting. PairDraft allocates prediction positions to schema-values and their end-of-value (EOV) markers, while the Pair State Machine (PSM) inserts known schema-keys. The Decoding State Encoder (DSE) identifies the current decoding position, and target context features provide schema information from the prompt with a one-shot example. Together, these mechanisms allow a single draft to cover multiple fields without predicting known structure.

  • PairCut: efficient schema-aware verification under concurrency. PairCut uses the determinism of injected schema-keys to extend verification prefixes when additional queries incur little latency. It preserves the baseline request widths and uses the available incremental budget to increase accepted progress and reduce subsequent decoding steps.


Additional Results and Clarifications

  • Schema injection ablations (R1 W2/Q2, R2 Q2). We added Pair-Schema-Aware DFlash, Schema-Aware DFlash, and Schema-Aware DFlash-SFT. At the same 16-token verification-width limit, DChord (16) achieves 29.1% higher average acceptance length than Schema-Aware DFlash and 11.8% higher than Schema-Aware DFlash-SFT. With full expanded verification width, DChord reaches 1.74× the average acceptance length of Schema-Aware DFlash-SFT. These comparisons support the contribution of compressed pair-level drafting beyond key injection and task adaptation.

  • Generalizability evaluations (R2 W1/Q1 and follow-up). We further evaluated DChord's generalizability across variable field ordering, pre-specified optional fields, unseen schemas, and longer field values. Across these settings, DChord achieves 1.15–2.92× acceptance length and 1.13–2.00× TPOT speedups over the corresponding DFlash baselines, demonstrating its effectiveness across diverse structured generation scenarios.

  • Task-adapted concurrency-serving comparison (R3 W1/Q1). We compared released DFlash, DFlash-SFT, and DChord on Qwen3.5-4B at concurrency 16 under both greedy and stochastic decoding. At the same verification-width limit, DChord improves throughput over DFlash-SFT by 11.20% at T=0T=0 and 10.01% at T=1T=1, averaged across the four datasets.

  • Further insights into PairCut (R1 W3, R2 W2/Q3). We explained its selection process with an example and compared execution statistics at concurrencies 16 and 32. On CUB dataset at concurrency 32, PairCut increases mean acceptance length from 22.81 to 25.25, reduces scheduler steps by 11.58%, and improves throughput by 9.82% over confidence-based pruning.

  • Further clarification of verification correctness (R3 W2/Q2). We clarified that DChord changes draft construction without altering the target's acceptance and rejection-correction rules or its output distribution under the same schema constraints. PairDraft uses predicted EOV solely to guide draft assembly, without submitting EOV to the target model for verification. We further clarified the reference distribution and bonus-token handling, demonstrating that DChord preserves the target model's output distribution under constrained decoding, for both greedy and stochastic sampling.


We have carefully addressed each individual comment provided by the reviewers and look forward to further discussion! We thank the reviewers again for their constructive feedback and will incorporate all new experimental results and clarifications into the revised manuscript.

Best regards, the authors of DChord

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.