acceptodds
Under review as a conference paper at ICLR 2027

Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting

Abstract

Block drafters propose several tokens in one forward pass, before earlier target tokens are realised. Their rejection mixes two losses: missing within-block path information and imperfect modelling of observable information. Accepted length cannot distinguish them. We separate the two with an information floor, the minimum expected rejection at a specified conditioning order; rejection above this floor is the model gap. Measurements across four domains and four open-weight targets, plus floor estimates on a frontier API target, yield three findings. First, the all-parallel floor reaches 0.286 at the final slot on Qwen3-4B, limiting even the best proposal to 71% per-slot acceptance. Second, one realised token removes 86–100% of this floor, a locality also recovered by an independent mutual-information analysis. Third, current drafters remain far above their floors: the final-slot model gap accounts for 43–64% of DFlash rejection and 85–92% of DSpark's oracle-conditioned rejection. Guided by this separation of short-range conditioning from proposal quality, we replace a context-independent predecessor correction with a prefix-attention head that reads committed context conditional on the predecessor. With supporting backbone components, the resulting Qwen3-4B drafter improves mean serving accepted length by 3.93% over the released DSpark checkpoint across nine tasks, without increasing the conditioning order. Code and data: https://anonymous.4open.science/r/specfloor-8CD4.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
speculative decoding, block drafting, LLM inference, information-theoretic analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_szC7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a new measurement framework for block drafters in speculative decoding, which decomposes the per-slot rejection risk into an information floor and a model gap. The authors find that for all-parallel drafters, e.g., DFlash, the information floor is nontrivial and limits the acceptance rate, whereas conditioning on a single realized predecessor, like the setting for DSpark, can substantially reduce it. Though a model gap still remains for this drafter. Motivated by these observations, the authors further propose a new prediction head called the "prefix-attention head" to replace the standard Markov head in DSpark. The proposed approach improves the average accepted length and gain speepdup.

Critical strengths

Pros

  1. The framework provides a useful perspective on distinguishing information limitations from the block drafters modeling capacity. The definition of the information floor is technically sound and provides a decomposition of rejection risk.

  2. The observation that conditioning on a single realized predecessor can remove most of the information floor is interesting, and the authors further support this finding through a conditional mutual information analysis.

  3. The experiments are board. The authors validate the floor-gap decomposition across multiple LLM families of varying sizes.

Critical weaknesses

Cons

  1. The analysis is mostly based on free-rollout risk, though the actual serving performance depends on the accepted length and rejection risk only on the reached path. Although the authors discussed them in Section 3.5, they did not give a comprehensive study on whether the claim and trend will still hold when decomposing serving risk. Given that there exists a substantial difference between these risks.
  2. Paper contains lots of models free-rollout gap decomposition, though the main table on average acceptance length only includes Qwen3-4B model. I would really like to see other model's performance e.g. larger models, different model family.
  3. I view this paper primarily as a dianostic paper, but I find the diagnostic analysis on the motivation of proposing prefix-attention head is somehow limited. There is indeed a huge model gap for DSpark, but it remains unclear whether the gap is actually caused by the Markov head or simply the drafter capacity issue. The authors provide some ablation studies but I would like to see more targeted experiments on which factors contribute most to the model gap.
Questions for authors

Some of the questions are already in the weakness sections.

  1. For DSpark, the proposed prefix-attention head indeed reduces the model gap, though the gap still remains. Could authors provide some discussion on the possible factors contributing the remaining gap?

  2. The experiemnts setup for free-rollout risk makes sense for me. However, since actual speculative decoding only reaches positions along previously accepted paths, why most of the conclusions are drawn from free-rollout risk rather than serving risk? Could the authors further clarify the practical significance of the proposed decomposition for actual serving?

  3. Could the authors extend the serving analysis to some larger model's e.g Qwen3-8B model? The additional serving results would help establish whether the observation could generalize. It would also be helpful to see whether the prefix-attention head will have similar improvment on larger models.

7 comments

Response to Reviewer szC7

Thank you for recognizing the value of the floor–gap decomposition and the finding that one predecessor removes most of the information floor. Your questions motivate two extensions: recomputing the decomposition on reached paths and extending the serving analysis to Qwen3-8B. Both support the central diagnostic findings. We also evaluate trained heads on Qwen3-8B, where prefix attention improves accepted length by 4.16%.

Does the diagnosis hold on reached paths

Yes. We now recompute both the information floor and drafter risk under the same survival-weighted distribution. For each slot, we weight target paths by their probability of surviving all preceding verification steps. We then optimize order-0 and order-1 proposals under these weights. Thus, the reached-path gap compares risk and floor on the same population.

The new measurements use 64 anchors across four domains and 256 target paths per anchor for each target. At the final slot:

TargetReached order-0 floorReached order-1 floorFloor removed by one predecessorDSpark reached risk
Qwen3-4B0.022660.0020391.1%0.17119
Qwen3-8B0.088910.0037695.8%0.10043

Floors are empirical weighted TV minima.

These results show that acceptance-induced path selection changes the numerical risks but preserves the central finding: one predecessor removes most of the information floor, while the current order-1 drafter remains substantially above that floor. Section 3.5 now summarizes this result, with the decomposition in Appendix F.2.

Why use free rollout and how does it guide serving improvements

Under the fixed context distribution, target rollouts make the information floor a property of the target and the information constraint. Conditioning on survival additionally makes it depend on the selected drafter. We report this complementary reached-path analysis above.

To connect the model-gap diagnosis to serving performance, Section 3.5 estimates how much accepted length can improve by optimizing one slot while keeping all other proposals fixed. We extend this analysis to Qwen3-8B:

Drafter on Qwen3-8BSlot-0 accepted-length gainSlot-6 accepted-length gain
DFlash0.2190.058
DSpark0.2830.049

For DFlash, the free-rollout model gap grows from 0.148 to 0.290 between slots 0 and 6, while the corresponding serving gain falls by about 3.8×. The final slot has the smallest gain in every domain for both drafters. This suggests prioritizing slots by their single-slot accepted-length gains: here, improving earlier slots offers greater serving benefit despite their smaller model gaps.

This emphasis on early-slot accuracy is consistent with later work, DMTP, which uses the target's pretrained MTP module for the first draft token and DFlash for the remaining parallel predictions.

Does the head improvement extend to Qwen3-8B

Yes. We trained drafters with Markov and prefix-attention heads on fresh Qwen3-8B-generated responses and evaluated them on 430 requests across nine tasks using their respective serving implementations:

HeadServing proposalMean accepted length
Markov headFull vocabulary3.5042
Prefix-attention headTop-16 lattice3.6501

Prefix attention improves accepted length by 4.16%, with positive gains on all nine tasks.

Both 8B variants train for five epochs with one seed each on the same 100k-conversation sample regenerated by Qwen3-8B, using a peak learning rate of 3e-4 and 104 warmup steps.

Why change the head rather than only increase capacity

We added a Markov head matched to prefix attention in parameter count. The following one-epoch results use the same 430 evaluation requests and top-16 lattice serving:

Head, with nomination lossMean accepted length
Markov head, rank 2564.5226
Markov head, rank 2854.5448
Prefix attention4.6448

The rank-285 Markov head has 86.60M parameters, within 0.14% of the prefix-attention head's 86.72M. Prefix attention exceeds this wider baseline by 0.100 accepted tokens, supporting the head design beyond increasing rank alone. Appendix H.7 reports this comparison and the head sizes.

What accounts for the remaining model gap

The information floor minimizes over all proposals with the specified information, whereas a trained drafter has finite capacity and a particular parameterization. Prefix attention removes one restriction but does not make that function class unrestricted. Other possible contributors include compression of the prefix into target and drafter features, finite data and optimization, and the mismatch between the combined training objectives and TV rejection risk. Candidate truncation can additionally affect the deployed proposal.

The design therefore tests a concrete opportunity revealed by the diagnosis: improving use of an already available predecessor and prefix can reduce rejection without increasing conditioning order. Section 4 now makes this connection explicit and discusses possible sources of the remaining gap.

Thank you to the authors for the detailed response and additional experiments. I appreciate the efforts to address my concerns. I do think the proposed decomposition is interesting and I think the authors have addressed most of my concerns.

However, I still find the practical impact of the proposed analysis somewhat limited. I initially expected the performance improvement to become more substantial on larger models, as decoding is already fast for small models. but the Qwen3-8B results still show a relatively small absolute gain in accepted length. Moreover, the extra analysis suggests that even the order-0 information floor is relatively low on reached paths. I maintain my score

Reviewer_wi9gOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

In speculative decoding, a drafter proposes multiple token continuations, and a larger model then verifies them. If a token continuation is rejected, this could be because (1) the drafter cannot see information from within the block, or (2) the drafter fails to fully utilize the information it observes. The paper’s central question is whether current drafters are close to their intrinsic limits—that is, whether they fully utilize all the information available to them. To distinguish between (1) and (2), the authors propose the following key metrics:

[1] Information Floor: The minimum rejection risk given the available information.

[2] Model Gap: The difference between the actual rejection risk and the minimum rejection risk.

[3] Exposure Difference: When the drafter can see prior tokens, how much error is caused by its modeling of those predecessor tokens.

The key empirical findings are:

[1] The information floor is large and grows with token depth, but it drops to a very low level once the drafter is given the previous token.

[2] The deployed drafter is still far from the information floor, so there is substantial room for improvement.

[3] This gap persists across model sizes, and there is evidence supporting this observation for frontier-scale models.

[4] Since access to one predecessor token removes most of the information floor, but the current order-1 drafter (a drafter that sees information about the previous token) remains far from this floor, the authors propose a new way to train the drafter. They show that it is faster and that the continuations accepted by the larger model are longer.

Confidence Declaration (to AC)

Dear AC: This paper is very far away from my own research expertise and I do not recall bidding any paper in this area. I believe I mostly understand the paper’s methodology, and did my best in giving reviews, but I had low confidence in my assessment because I lacked the background knowledge needed to judge how important/novel this work is relative to its own academic community. Please take this into consideration when making the decision.

Critical strengths

I think the paper is well written overall, (except for Section 4, which I personally find a bit hard to follow). The metrics are clearly defined in Section 2, and Section 3 does a good job evaluating existing drafters using these metrics. I like the paragraph in Section 3.4 where the authors provide evidence that their findings also apply to frontier-scale models.

Critical weaknesses

[1] In Section 4, the improvements in the total gap (Figure 7) and expected acceptance length (Table 1) are relatively small. Are there any error bars showing that these improvements are statistically meaningful?

[2] The implementation and training are described too briefly in Section 4, making the exact details hard to understand.

Questions for authors

[1] My understanding is that Section 4 argues that the correction should be a function of both the predecessor token and the prior context. How is this motivated by the earlier findings? Is the sole reason that prior sections find that DSpark does not leverage the information of predecessor token optimally. I think writing-wise this section seems a bit disconnected from the preceding sections.

[2] Is there a reason why the exposure difference (Figure 7b) drops with the proposed prefix-attention head?

3 comments

Response to Reviewer wi9g

Thank you for the careful reading and for highlighting the clarity of the measurement framework and the broader model evidence. We address the uncertainty of the Section 4 improvements, clarify how the diagnostic findings motivate the head, and give a more explicit account of implementation and training. We also clarify the direction of the exposure difference in Figure 7.

Statistical uncertainty in Table 1 and Figure 7

We have added paired prompt-cluster bootstrap intervals for the accepted-length evaluation. At temperature 1, mean accepted length improves by 3.93% (95% CI [3.61%, 4.23%]). At temperature 0, the relative improvement is 2.89% [2.65%, 3.13%].

For Figure 7, we resample the 170 prompt clusters containing the 384 evaluation anchors, preserving the original weighting and pairing at the prompt level. The final-slot results are:

Quantity at slot 6DSparkOursOurs minus DSpark with pointwise 95% CI
Model gap0.32540.2424-0.0830 [-0.1173, -0.0488]
Exposure difference0.21430.2615+0.0471 [+0.0223, +0.0734]
Total gap0.53970.5039-0.0358 [-0.0659, -0.0048]

Here total gap is model gap plus exposure difference, as plotted in Figure 7. These intervals quantify prompt-level variability for the fixed checkpoints and recorded rollouts. They support the final-slot reductions in both model gap and total gap. The revision includes intervals in Figure 7 and Table 1, with the paired gap changes in Table 15.

How the diagnostic findings motivate prefix attention

The diagnosis establishes two facts: one realized predecessor removes most of the information floor, yet DSpark remains far above that order-1 floor. Together, these suggest improving the use of the information already available at order 1.

The specific design follows from examining DSpark's Markov head. Its logits are uk(X)+BA[a]u_k(X)+BA[a], where XX is the committed prefix and aa is the predecessor. Although its normalized prediction depends on both inputs, changing the predecessor from a′a' to aa changes the log odds of successors vv and ww by (Bv−Bw)(A[a]−A[a′])(B_v-B_w)(A[a]-A[a']), independently of XX. Thus, the predecessor-induced correction to relative preference cannot adapt to the prefix.

Prefix attention removes this restriction: the predecessor and drafter state form a query that reads committed-prefix features. The same predecessor can consequently guide different corrections in different contexts. The head still observes only the prefix and one predecessor, so its information floor is unchanged under the same evaluation law. The measured gap reduction tests whether this richer use of the available information helps. Section 4 now opens with this connection between the diagnosis and the head design.

Implementation and training details

For each speculative position, the five-layer parallel drafter produces a state and base logits. The head separately normalizes the state and frozen target embedding of the predecessor, concatenates them, and projects the result to width 512. It then applies prefix cross-attention with four heads of dimension 128 and a SwiGLU block of hidden width 2048. Its vocabulary projection supplies a correction to the base logits. Candidate predecessor tokens share projected prefix keys and values, while each query reads the prefix independently.

At inference, the base logits nominate 16 candidates per position. The head scores candidate transitions across positions in parallel, and a short lattice walk selects the draft chain. Conditional scores are normalized over nominated successors, and verification uses that same proposal distribution.

The 4B model uses 1,339,867 Qwen3-4B-regenerated Open PerfectBlend conversations, a maximum sequence length of 4096, and 512 anchors per sequence. We train for ten epochs, or 26,160 steps, with batch size 512, learning rate 6e-4, 4% warmup, and cosine decay. The target embedding and output tables are frozen. We retain DSpark's objectives and add candidate-nomination loss. Section 4 now summarizes the implementation and training; Appendices H.1–H.2 give the full parameterization and loss definitions.

Direction and interpretation of the exposure difference

The exposure difference increases in Figure 7b. It is defined as self-conditioned risk minus oracle-conditioned risk. At the final slot, both risks decrease, but oracle-conditioned risk decreases more, so their difference grows.

The result supports better prediction when the correct predecessor is available, while the improvement is smaller with self-generated predecessors. The revised Figure 7 caption and Section 4 make this direction explicit.

Reviewer_wi9gOpenReview ↗

Acknowledging Rebuttal

Thank you for answering my questions, and clarify the implementation detail in Section 4 and Appendix H.1–H.2. I don't have further questions, and plan to keep my original score (3: Weak Accept).

Thank you for your time and for considering our responses. We appreciate your feedback!

Reviewer_hu8oOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks into speculative decoding with block drafters. Block drafters are small models that guess several next tokens at once for a large LLM to check. Because all guesses are made at once in block drafters, each guess is made without knowing the tokens before it, so some rejections are unavoidable. The paper splits a drafter's rejections into an unavoidable part (which they call as "information floor") and a fixable part (the "model gap"), and measures both across four domains and four open models. The paper finds that unavoidable mistakes grow the further a guess is within a block, and that letting each guess see just the one token before it removes most of them. Based on this, the paper proposes a prefix-attention head that makes better use of the previous token and earlier text, giving a modest improvement in accepted length over the existing DSpark drafter on Qwen3-4B.

Critical strengths

-Usually you only see how many guesses a drafter gets right, which can't tell you how close it is to the best possible drafter given the same information. This paper measures that limit, and the paper checks that measurement several ways. -The paper shares a strong, useful finding i.e., seeing just one previous token removes most of the unavoidable mistakes. This holds across several models.

Critical weaknesses

-The improvement from the new method is small. The gain over DSpark is modest: from 4.75 to 4.94 accepted tokens (Table 1). -No variation across seeds is reported. -Several changes are done together: prefix-attention head, two-tap convolution, slot embeddings, candidate-nomination loss, DFlash2's search, etc. So it's unclear which change caused the gain. The short ablation in the Appendix doesn't separate the head from the new loss, so it's unclear how much of the gain comes from the head itself.

Questions for authors

Was your drafter trained on the same data and compute as the released DSpark checkpoint you compare against? Appendix describes your training setup but doesn't say whether it matches DSpark's. Also, can you report results over a few seeds?

3 comments

Response to Reviewer hu8o

Thank you for highlighting the value of separating information limitations from model error and the finding that one predecessor removes most of the information floor. We have added statistical analyses, repeated evaluations, and more targeted head comparisons to address your concerns about the magnitude, reproducibility, and attribution of the improvement.

Magnitude of the gain and variation across seeds

The accepted-length improvement is consistent across repeated evaluations and translates into a measurable throughput benefit. A paired prompt-cluster bootstrap of the original Table 1 evaluation gives:

TemperatureDSpark accepted lengthOursRelative gain with 95% CI
05.16915.3183+2.89% [2.65%, 3.13%]
14.75324.9398+3.93% [3.61%, 4.23%]

We also evaluated both checkpoints with six generation seeds at temperature 1:

Generation seedDSpark accepted lengthOursRelative gain
9804064.75004.9323+3.84%
14.74574.9491+4.29%
24.75324.9243+3.60%
34.74184.9427+4.24%
44.73904.9326+4.09%
54.74764.9446+4.15%

The mean relative improvement is 4.03%, with a standard deviation of 0.26 percentage points across generation seeds.

To check whether the heavier head delivers an actual runtime benefit, we additionally ran six pairs of fresh server launches on one H100. On a fixed 430-request subset at temperature 1 and concurrency 1, the mean task-macro throughput improvement over DSpark is 4.96% (95% t interval [3.91%, 6.01%]). This repeat study supports a practical benefit.

Separating the head from nomination loss and other components

You are right that the original head ablation also changed the nomination objective. We trained the Markov head with nomination loss and re-evaluated the relevant one-epoch checkpoints in the same runtime. All compared models exclude short convolution and slot embeddings:

ConfigurationServing proposalMean accepted length
Markov headFull vocabulary4.5093
Markov head + nominationFull vocabulary4.5257
Markov head + nominationTop-16 lattice4.5226
Prefix-attention head + nominationTop-16 lattice4.6448

Adding nomination loss or switching to lattice serving changes the Markov result only slightly. With nomination loss and lattice serving shared, prefix attention gains 0.122 accepted tokens over Markov (95% CI [0.102, 0.142]). Table 17 separately evaluates short convolution and slot embeddings.

Training data and compute relative to released DSpark

Yes, we follow DSpark's public data-generation recipe and training schedule. Both use target-regenerated Open PerfectBlend conversations, five drafter layers, block size 7, batch size 512, learning rate 6e-4, 4% warmup, and ten training epochs.

We regenerate every assistant turn using Qwen3-4B at temperature 0.7, top-p 0.8, and top-k 20, with thinking disabled. After tokenization and filtering, the training set contains 1,339,867 conversations and training takes 26,160 optimizer steps.

Section 4 and Appendix H.2 now make this training comparison explicit.

Thanks for addressing the comments and questions. If accepted, please make sure to include these in the final version. Nice to see that the prefix-attention head itself gives a gain of 0.122 accepted tokens. I see that the head-isolation ablation is on a one epoch checkpoint only, it would have been useful to see if the full 10 epoch model shows the same ordering. Given that rest of my questions/comments are addressed, I'm increasing my score to 3.

Thank you for your positive feedback and for reconsidering your score. We appreciate your thoughtful comments and the time you spent reviewing our work. We will make sure to incorporate the additional ablation results into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.