acceptodds
Withdrawn from ICLR 2027

Beyond the Tip of the Iceberg: Depth-Aware Top- Margin Distillation for Diffusion Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding accelerates LLM inference by using a lightweight drafter to propose future tokens for parallel verification. Diffusion-based and block-parallel drafters can generate long speculative sequences efficiently, but their acceptance often degrades rapidly with speculative depth, particularly at longer context lengths. We introduce Depth-Aware Top-K Margin Distillation, a training objective designed to improve drafter–verifier alignment by accounting for the unequal importance of speculative positions. At each depth, our objective distills the verifier's distribution over its Top-K candidate tokens and aligns the logit margins between adjacent ranked candidates, while assigning larger weights to earlier positions whose rejection invalidates the remaining speculative suffix. The formulation requires no modification to the drafter architecture or inference procedure and can, in principle, be applied to different speculative drafter families, while being particularly well suited to diffusion and parallel drafting where depth-wise acceptance degradation is pronounced. We apply our objective to a block-diffusion drafter with three target models (Qwen3-8B, Qwen3-14B, and Gemma-4-31B) on five benchmarks. With Qwen3-8B and Gemma-4-31B, it improves the average acceptance length by 15–16% over the same drafter trained with its original objective. With Qwen3-14B, it improves acceptance length by 28% over a published baseline that uses a smaller proposal budget. These gains come with correspondingly higher end-to-end speedup. The benefit grows with context length: on BABILong, the relative gain in acceptance length rises from 28% at 8K tokens to 55% at 1M tokens. These gains come with no additional inference-time overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, Large language models, Depth-aware loss

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uVwZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DTopK, a training objective intended to improve drafter–verifier alignment without changing the drafting architecture or inference procedure. It combines depth-weighted hard-label cross-entropy, distillation over the verifier’s Top-K candidates, and squared matching of adjacent-rank logit margins. Earlier speculative positions receive larger weights because their acceptance enables the remaining prefix to survive verification. The analysis derives an exponential weighting interpretation under constant conditional acceptance and establishes the equivalence between exact margin matching and matching the restricted Top-K distribution. Experiments apply DTopK to DFlash with three verifiers across five standard benchmarks and a BABILong context-length sweep, reporting approximately 15–16% higher mean committed tokens per cycle on Qwen3-8B and Gemma-4-31B.

Critical strengths
  1. The proposed modification is general and straightforward to integrate. DTopK changes the training objective without introducing additional model parameters or inference-time operations. This is an attractive direction for improving an existing speculative-decoding implementation without redesigning its runtime.

  2. The proposed method achieves improvements across multiple benchmarks. Table 1 reports higher acceptance length and speedup than DFlash for all fifteen verifier–benchmark combinations.

  3. The paper provides some interpretable and insightful analysis. The prefix-survival identity correctly distinguishes conditional per-depth acceptance from accepted-prefix length without requiring positional independence. The per-depth measurements also attempt to connect the observed gains with the proposed motivation.

Critical weaknesses
  1. Current evidence does not establish that the distinctive margin-based component is responsible for the gains. Missing component ablations and numerical hyper-parameters prevent a reliable assessment of the incremental contribution.

  2. The reported experimental results are inconsistent in Figure 1a and Table 2. The source code is not provided, and the implementation details remain unclear, which raises concerns regarding the reliability of the results.

Questions for authors
  1. The component ablation establishing the contribution of the central margin-matching term is missing. The experiments compare the complete objective against DFlash but do not isolate Top-K distillation, margin matching, or the choice of depth weighting; this is particularly consequential because Appendix A.1 establishes that the two distillation terms share their minimizers.
  2. Some implementation details and design choices of the proposed objective are missing. The manuscript does not state the actual values of KK, τKD\tau_\mathrm{KD}, λTopK\lambda_\mathrm{TopK}, or λMargin\lambda_\mathrm{Margin}, and the numerical discussion of γdepth\gamma_\mathrm{depth} concerns illustrative values and the original DFlash setting rather than an explicit configuration for each experiment.
  3. The assumption that the selected Top-K candidates capture the acceptance-relevant information is not validated. Eqs (8)–(12) match distributions and margins after restricting and renormalizing to the teacher-selected set, so they do not themselves control the full-vocabulary probability overlap used in the stochastic-acceptance expression. The authors acknowledge this but rely on hard-label CE without measuring the remaining mismatch.
  4. Figure 1a shows Orthrus at ~3.8 committed tokens per cycle and DFlash at ~1.45 at 64K context, whereas Table 2 reports 2.70 and 1.80 for the same verifier (i.e., Qwen3-8B) and nominal context length. The authors should clarify such performance inconsistency.
  5. Table 2 reports acceptance length but no long-context throughput or latency, so reaching 1.70 committed tokens per cycle at 1M context does not establish a speedup over autoregressive decoding once drafting and verification costs are included. The evaluation also relies on unspecified RoPE-based context scaling.
Reviewer_74HJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that diffusion and block-parallel speculative drafters accept far fewer tokens as speculative depth and context length grow, because an early rejection throws away the rest of the draft. It proposes Depth-Aware Top-K Margin Distillation, which trains the drafter to match the verifier’s Top-K candidates and their logit margins, with higher weight on earlier positions. The same drafter then accepts 15–16% more tokens on average, and up to 55% more at 1M-token context, with no extra inference cost.

Critical strengths
  1. Precise and under-explored problem formulation.
  2. The idea makes sense.
Critical weaknesses
  1. The writing is very poor. The introduction is too long. It is strongly recommended that the author split it into two sections: "Introduction" and "Related Work".
  2. There is a lack of sufficient ablation experiments to prove the effectiveness. Firstly, depth weighting already exists. Then, Top-K distillation also has precedents. The author added the adjacent-rank margin here, but there is no experimental evidence to show that it is better than simple Top-K KD.
  3. In addition, please do not directly quote the direct speed measurement data from dspark/DFlash. Many times, some minor details can cause the working conditions to be completely different. It must be retested by yourself.
Questions for authors

How does this method perform on the dflash2?

Reviewer_2etDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the collapse of deep speculative acceptance for parallel/diffusion drafters in long-context speculative decoding. It proposes DTopK, which supervises the drafter at each speculative depth using the verifier’s Top-KK candidate distribution and adjacent-rank logit margins, sharing a depth-decay weight across all loss terms. The method leaves the drafter architecture, verification rule, and inference cost unchanged. The authors apply it to DFlash and report acceptance-length and speedup gains on Qwen3-8B, Qwen3-14B, and Gemma-4-31B.

Critical strengths
  1. The problem is important. The paper clearly characterizes the unequal value of speculative depths and the acceptance collapse of parallel drafters in long contexts using the prefix-gated formula.
  2. The long-context results are compelling. Relative to DFlash, DTopK improves acceptance length from +28.0% at 8K to +54.5% at 1M tokens; at 1M tokens, DFlash accepts only 0.10 draft tokens per cycle, while DTopK accepts 0.70.
Critical weaknesses
  1. The individual contributions of the three loss terms are not isolated, and the role of the margin term is unclear.
    DTopK combines three loss terms: depth-weighted cross-entropy, top-KK distribution distillation, and adjacent-rank margin matching. The first two closely follow existing work, and there is no ablation isolating the contribution of the third term. Moreover, the paper itself states in A.1 that the second and third share the same convergence point, leaving the benefit of the margin term unclear.

  2. Mismatch between motivation, method, and evaluation.
    The motivation is centered on long-context degradation, but the proposed method is not inherently long-context-specific, and the evaluation does not isolate whether the gains come from long-context handling.

  3. Missing training details.
    It is unclear whether the drafter is trained from scratch or fine-tuned, what training dataset is used, or the optimizer, learning rate, batch size, number of training steps, learning-rate schedule, and other optimization hyperparameters.

  4. Unfair comparison on Qwen3-14B.
    On Qwen3-14B, the EAGLE-3, DFlash, and DSpark rows are taken from the published results of Cheng et al. (2026). Performance gains are hardware-dependent, so speedup should be compared under the same hardware configuration rather than across published numbers from different setups.

Questions for authors
  1. What does the adjacent-rank margin matching term add beyond top-KK distillation? The paper states that these two terms share the same convergence point; if so, the unique benefit of margin matching remains unclear. How does this signal differ from prior objectives such as HASS?
  2. Provide ablations that decompose the three loss terms. Which gains are attributable to depth-weighted cross-entropy, top-KK distillation, and margin matching, individually and in combination? The current evaluation only compares full DTopK against DFlash.
  3. Clarify the relationship between DTopK and Orthrus. DTopK underperforms Orthrus in several configurations; under what conditions does Orthrus remain stronger, and why does the paper not explain this?
  4. Specify the training setup. Is the drafter trained from scratch or initialized from a pretrained checkpoint? What training data and hyperparameters are used? These details are currently missing and limit reproducibility.
  5. The experiments are limited to full-attention dense models such as Qwen3. Can the authors evaluate DTopK on MoE or linear-attention models as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.