acceptodds
Desk rejected at ICLR 2027

BagDraft: Next-Bag-of-Tokens Drafting for Lossless Speculative Decoding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The speedup of speculative decoding is fundamentally bounded by draft token throughput, yet existing drafters either incur sequential passes (EAGLE), dilute model capacity across disjoint heads (Medusa), or tie capacity to rigid per-slot block predictions (DFlash). We propose BagDraft, which reformulates speculative drafting as a next-bag-of-tokens prediction task. Crucially, the informational debt of discarding sequence order requires no parametric recovery in speculative decoding, as exact target verification guarantees lossless generation. Operating via a single forward pass with a shared logit vector, BagDraft directly models future -token sets under a multi-hot cross-entropy objective, using discrete anchor blocks to ensure strict input isomorphism between training and inference. At inference, we mitigate the permutation explosion by formalizing path probabilities via Plackett–Luce sampling without replacement; its proven prefix-monotonicity property mathematically justifies expectation-based pruning to construct compact permutation trees, verified by the target model in a single forward pass. Across 7 benchmarks on Qwen3-4B/8B, BagDraft achieves an average acceptance length of and a wall-clock speedup on 4B ( over DFlash, over EAGLE-3), proving that speculative drafters can surpass state-of-the-art efficiency without dedicated per-position heads or autoregressive rollout, relying solely on an offset-decayed shared distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining, speculative decoding

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_41MFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BagDraft uses a lightweight draft model to predict an unordered bag of likely upcoming tokens in a single forward pass, then organizes these tokens into a candidate tree for parallel verification by the target model. This shifts part of the burden of modeling token order from the drafter to inference-time search and verification, while preserving the target model’s output distribution.

Critical strengths
  1. Decoupling sequential dependencies from the draft model’s prediction task is a promising idea.
Critical weaknesses
  1. I do not see a substantial performance gain from decoupling sequential dependencies. The paper also lacks comparisons with state-of-the-art parallel drafters such as DSpark. I would not expect the proposed method to outperform DSpark, which incorporates an autoregressive (AR) head.

  2. The reliance on inference-time candidate-tree construction and tree attention also limits the method’s practicality. Production LLM serving systems almost exclusively use chain-structured drafts, which DFlash and DSpark support natively. Consequently, in terms of both expected performance and deployment feasibility, the proposed approach appears less competitive than state-of-the-art speculative decoding methods such as DSpark.

Questions for authors
  1. Could you provide a figure comparing your method with DSpark on Qwen3-4B and Qwen3-8B? Official DSpark checkpoints are available for both target models.

  2. If we take the union of the top-kk tokens from DFlash’s predicted logits at each of its mm positions, yielding up to m×km \times k unique tokens, this also forms what you call a “next bag of tokens.” What fundamentally distinguishes your method from this construction? An empirical comparison would help clarify the distinction.

Reviewer_yKw2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The core idea of ​​the paper is clear: predict the future token set using a shared distribution, and then verify the restored order through permutation trees and a target model.

  • Proposes BagDraft, a speculative decoding framework that predicts future tokens using a single shared distribution instead of position-specific predictions.

  • Trains the drafter with an offset-weighted multi-hot cross-entropy objective and isolated anchor queries.

  • Constructs candidate trees using Plackett–Luce path scores and beam pruning, followed by target-model verification that preserves the output distribution.

  • Reports average speedups of 3.168× on Qwen3-4B and 2.987× on Qwen3-8B across seven benchmarks, improving over DFlash by 12.1% and 5.2%, respectively.

Critical strengths
  • The main idea is simple and interesting: predict which tokens will appear next, then let tree search and target verification determine their order.
  • The reported gains over DFlash are encouraging, especially on dialogue tasks. BagDraft is faster on all seven benchmarks with Qwen3-4B.
  • The analysis explains what the shared distribution learns and gives a clear argument for why verification preserves the target distribution.
Critical weaknesses
  • The experiments do not isolate the main contribution. BagDraft changes the training objective, query structure, tree construction, and verification implementation together. The reported gains therefore do not show how much comes from predicting a bag of tokens. A controlled comparison with position-specific prediction under the same backbone, training budget, and inference implementation is missing.

  • The claimed training–inference equivalence needs clarification. Equation (9) allows an anchor query to access the target hidden state at that anchor. However, under the decoding procedure in Section 3.5, the last committed token is a newly sampled bonus token whose target hidden state is not yet available. The paper should explain whether features are shifted or an additional target forward pass is required. This directly affects a central methodological claim.

  • The pruning argument does not establish the claimed connection to expected acceptance. Prefix monotonicity holds for the assumed Plackett–Luce distribution, but the relevant quantity is the probability of reaching a path under the target model. The shared bag distribution does not retain token dependencies, and the paper provides no evidence that its path scores approximate these probabilities well. The theoretical justification for the pruning strategy is consequently weaker than presented.

  • Several essential ablations are missing. In particular, the paper does not compare uniform bag supervision and next-token-only training while holding the inference tree fixed. Since approximately 87% of the default loss weight falls on the first two offsets, the benefit of learning the broader bag remains unclear. The contributions of root pinning, parent widening, repeated-token leaves, and selective LM-head evaluation are also not separated.

  • The latency analysis is incomplete and internally inconsistent. Section 5.4 attributes the reduced gains on 8B to increased verification cost, whereas Appendix A.8 states that target-forward latency is largely unchanged and identifies the drafter as the bottleneck. The appendix does not provide the numerical stage-level breakdown promised in the main text. Baseline block sizes, tree budgets, and timing variability also need to be reported to make the speed comparisons convincing.

  • The novelty of the anchor-block training scheme is overstated. DFlash already uses randomly sampled anchors, isolated training blocks, and exponentially decaying loss weights. Reducing this construction to a single query is a reasonable adaptation, but the paper does not clearly distinguish that adaptation from the existing training framework.

  • Some quantitative claims need reconciliation. The acceptance rates in Table 2 imply a mean acceptance length of 3.986, compared with 3.976 in Table 1; the discrepancy exceeds what rounding the displayed rates would explain. Similarly, Appendix A.6 motivates the five-layer drafter using training coverage alone, without showing whether its additional inference cost improves end-to-end speed. These gaps weaken confidence in the analysis supporting the final configuration.

Questions for authors
  1. In Equation (9), does the anchor query attend to the target hidden state of the anchor token itself? If so, how is that state available at inference when the anchor is the bonus token from the previous step?

  2. How does BagDraft compare with position-specific prediction when the backbone, training data, training budget, and verification implementation are matched?

  3. With the inference tree held fixed, what happens when the drafter is trained with uniform bag supervision or next-token-only supervision? How much does each tree-expansion heuristic contribute to the final speedup?

  4. How well do the Plackett–Luce path scores correlate with actual target-model reach probabilities? Do they outperform simpler scoring rules under the same verification budget?

  5. Can you provide a numerical latency breakdown and the decoding configurations used for each baseline? How should readers reconcile the different bottleneck explanations in Section 5.4 and Appendix A.8?

  6. Why do the per-offset rates in Table 2 imply an acceptance length of 3.986 rather than the 3.976 reported in Table 1? Were different runs or aggregation procedures used?

Reviewer_RBtMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BagDraft, a speculative decoding method that predicts a bag of future tokens in a single forward pass using a shared vocabulary distribution. The predicted tokens are organized into a pruned permutation tree and verified by the target model without altering its output distribution. On seven benchmarks, BagDraft achieves average speedups of 3.17x on Qwen3-4B and 2.99x on Qwen3-8B relative to autoregressive decoding at batch size 1 and temperature 1.

Critical strengths
  1. The paper is well written and clearly explains the motivation behind BagDraft.
  2. The experiments are presented clearly, with detailed comparisons across seven benchmarks and two model sizes.
  3. The architecture is simple and easy to follow. The use of independent anchor masking keeps training consistent with the single-query setup used during inference.
Critical weaknesses
  1. The paper should clarify its contribution relative to prior work on future-token prediction [A] and parallel drafting [B, C]. More controlled comparisons would make it easier to distinguish the benefits of bag prediction from those of the training procedure and search strategy.
  2. The limitations of predicting an unordered token bag are not examined in sufficient detail. An analysis of ordering errors and repeated tokens would help explain when and why the method produces shorter accepted drafts.
  3. The evaluation is limited to a single model family and a narrow range of decoding settings. Experiments with additional model families, decoding configurations, and repeated runs would provide stronger evidence that the reported speedups are consistent.
  4. The efficiency analysis does not clearly distinguish the costs of drafting, tree search, and target-model verification. As a result, it is difficult to identify the main bottleneck or assess how the method scales to larger models and different search budgets.

[A]. DBeyond Multi-Token Prediction: Pretraining LLMs with Future Summaries. ICLR, 2026.

[B]. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML, 2024.

[C]. DFlash: Block Diffusion for Flash Speculative Decoding. ICML, 2026.

Questions for authors

See Above.

Reviewer_8GSfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper predicts multiple future tokens using a shared logit vector and is trained with an offset-weighted bag objective. During inference, it constructs a permutation tree via Plackett–Luce scoring, followed by lossless verification through target sampling. On seven benchmarks—evaluated using Qwen3-4B/8B models with a temperature of 1—the paper reports an average speedup of 3.168×/2.987× over autoregressive decoding, representing an improvement of approximately 12.1%/5.2% over DFlash.

Critical strengths
  1. The use of a single shared distribution to provide candidates for multiple tokens—delegating part of the sequential prediction process to tree construction—is a meaningful design for a drafting mechanism. The paper describes the distribution fitted via a weighted objective and provides a proof that target sampling preserves the output distribution under the stated conditions, establishing a solid methodological foundation for this approach.
  2. The experiments report both acceptance length and wall-clock speedup across mathematical, coding, and conversational tasks. On a 4B model, the method outperforms two reported baselines across seven benchmarks, and it also shows average gains on an 8B model. These results support the feasibility and potential for practical efficiency gains offered by the proposed method.
Critical weaknesses
  1. The efficiency comparison fails to include a tree-based baseline directly relevant to the paper’s core claims. While the paper mentions DDTree in Section 2—a method capable of constructing trees from DFlash’s position-wise predictions—this comparison is absent from Table 1, and the actual number of nodes verified by each method is not clearly specified. Furthermore, although Section 5.4 refers to a stage-level latency breakdown in Appendix A.8, the appendix itself does not provide the corresponding granular data. Consequently, the current results are insufficient to determine whether adopting a "shared bag representation" offers a superior efficiency trade-off compared to the existing approach of combining position-wise drafting with tree expansion.
  2. The incremental benefit of the "bag objective" itself has not been isolated. Existing ablation studies on bag size and backbone depth do not address the following: if the backbone, training data, and permutation-tree inference remain constant, what would be the performance if the training objective were shifted to "next-token-only" supervision? Theoretical analysis of the fitted distribution cannot serve as a substitute for this empirical validation; thus, it remains unclear to what extent the multi-position "bag supervision" contributes to the acceptance rate and the final speedup.
Questions for authors
  1. Does the DFlash baseline in Table 1 employ tree expansion? When using comparable verification budgets—and after applying the LM-head optimization described in Section 3.5 to the relevant baselines—does BagDraft still offer a latency advantage or a superior efficiency trade-off compared to DFlash+DDTree? Providing the actual number of verification nodes and the stage-level timings mentioned in Section 5.4 would help clarify this comparison.
  2. Is there a controlled ablation—keeping the backbone, training data, training budget, and inference settings constant—that compares the offset-weighted bag objective against next-token-only supervision? If so, how do the acceptance length and wall-clock speedup change? This comparison must retain the original multi-layer permutation tree to effectively disentangle the gains derived from the supervision objective from those provided by tree expansion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.