acceptodds
Under review as a conference paper at ICLR 2027

LatentSpec: Two-Stage Thinker–Drafter Parallel Drafting with Shared Latent Reasoning for Speculative Decoding

Abstract

Speculative decoding accelerates LLM inference by using a lightweight draft model to generate multiple candidate tokens, which are then verified in parallel by the target model. Recent parallel drafting methods reduce drafting latency by predicting an entire candidate block in a single forward pass. However, although existing methods allow information exchange across candidate positions, they typically lack an explicit shared block-level representation of the future, making it difficult to globally coordinate candidate-block generation and potentially limiting both the overall quality of the candidate block and the length of its accepted prefix. Inspired by recent advances in latent reasoning, we propose LatentSpec, a parallel drafting model with a two-stage Thinker–Drafter architecture. The Thinker first updates a small set of latent queries using intermediate features from the frozen target model, thereby constructing a shared latent reasoning state in a continuous hidden space. Conditioned on this state, the Drafter then coordinates and predicts the candidate block as a whole in parallel within a single forward pass. Experiments on multiple benchmarks with Qwen3 models show that, while preserving or improving average acceptance length relative to DFlash, LatentSpec reduces drafting FLOPs by 8.26–9.41% and increases end-to-end throughput by up to 9.33%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Speculative Decoding, Latent Reasoning, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hjWaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes LatentSpec, a two-stage Thinker–Drafter architecture for parallel speculative decoding. The Thinker processes a small set of latent slots conditioned on target-model features, and the Drafter uses the resulting shared representation to predict a candidate block in parallel. By allocating part of the computation to fewer positions, the method reduces drafting FLOPs while maintaining acceptance lengths comparable to DFlash. Experiments on Qwen3-4B and Qwen3-8B report modest end-to-end throughput improvements.

Critical strengths
  1. The architecture offers a reasonable approach to allocating computation within parallel drafters. It retains parallel token prediction and standard lossless verification without introducing additional target-model forward passes. The reported results suggest that some full-block computation can be replaced with computation over fewer latent positions while largely preserving draft quality.

  2. The evaluation includes practical serving measurements. The authors retrain baselines on the same target-regenerated data and separately report analytical FLOPs, draft latency, and end-to-end SGLang throughput across multiple request concurrencies. This provides useful evidence beyond acceptance length alone.

Critical weaknesses
  1. The experimental evidence is insufficient to establish a robust efficiency advantage. The reported comparisons use first-epoch checkpoints without convergence curves, whereas DFlash and DSpark report training for 6 and 10 epochs, respectively. This matters because the acceptance-length differences are small. The throughput comparison also omits the DFlash-3L baseline and both DSpark and DSpark+LatentSpec, leaving the efficiency trade-off against simpler or stronger alternatives unresolved. Finally, throughput evaluation is restricted to two mathematics datasets with thinking disabled and a maximum of 512 generated tokens. Longer-training comparisons and longer-generation measurements are needed to determine whether the reported gains persist.

  2. The distinctive contribution of the latent mechanism is not sufficiently established. The architecture provides a reasonable computational reallocation strategy, but the evidence for the stronger claim that latent reasoning coordinates the candidate block remains limited. The main ablation against DFlash-3L introduces an additional two-layer Thinker, changing both capacity and computation, while acceptance-length differences against DFlash-5L are small and not consistently positive. More controlled comparisons against simpler alternatives with similar computational budgets are needed to isolate the benefit of the proposed shared latent structure.

Questions for authors
  1. How do acceptance length and end-to-end throughput evolve beyond the first training epoch, such as 6 epochs? Do the relative advantages persist once all compared methods have converged?

  2. How does LatentSpec compare with simpler DFlash variants at similar computational budgets? In particular, what is the end-to-end throughput of the DFlash-3L baseline?

  3. What are the measured latency and throughput of DSpark+LatentSpec relative to DSpark? Do the gains persist with generation limits of 2,048 or 4,096 tokens, and what are the actual output lengths and truncation rates?

4 comments

We thank the reviewer for the careful review and constructive feedback. Our detailed responses follow.

Q1. How do acceptance length and end-to-end throughput evolve beyond the first training epoch, such as 6 epochs? Do the relative advantages persist once all compared methods have converged?

A1. Given our limited computational resources, we trained EAGLE3, DFlash (5 layers), and LatentSpec for four epochs and observed convergence of the loss curves. The results in Table 1 show that LatentSpec achieves higher average acceptance lengths than both DFlash and EAGLE3 after multiple training epochs. In particular, LatentSpec reduces training and inference FLOPs by 8.26%–9.41% relative to DFlash at context lengths of 512–4096, while achieving higher average acceptance lengths through a more effective allocation of computation in its two-stage architecture.

Table 1: Average acceptance length τ on Qwen3-4B for EAGLE3, DFlash and LatentSpec after 4 epochs under sampling (T = 1).

Draft ModelMATH-500 (Math)AIME25 (Math)HumanEval (Code)MBPP (Code)MT-Bench (Chat)Alpaca (Chat)Overall Avg
Temperature T = 1 (sampling)
EAGLE34.373.623.913.472.292.113.30
DFlash5.034.194.834.533.163.024.13
LatentSpec5.054.254.904.583.193.044.17

Q2. How does LatentSpec compare with simpler DFlash variants at similar computational budgets? In particular, what is the end-to-end throughput of the DFlash-3L baseline?

A2. To compare methods under similar computational budgets, we evaluated LatentSpec and DFlash-4L. Relative to DFlash-5L, LatentSpec and DFlash-4L reduce FLOPs by 8.26%–9.41% and 11.32%–12.45%, respectively, at context lengths of 512–4096. As discussed in EAGLE-4 [1], the LM head accounts for a large proportion of the draft model’s computation, so DFlash-4L does not reduce computation by nearly 20% relative to DFlash-5L. Since the two methods achieve similar reductions relative to the same baseline, we use them for a comparison under similar computational budgets. We also measured the end-to-end throughput of DFlash-3L, as shown in Table 2. With its higher average acceptance length and lower FLOPs than DFlash-5L, LatentSpec achieves higher throughput than DFlash-3L, DFlash-4L, and DFlash-5L in most configurations. The average acceptance lengths of DFlash-4L are reported in Table 6.

Table 2: End-to-end aggregate throughput (tokens/s) on Code (HumanEval, MBPP) with a fixed output length of 512 tokens.

DatasetMethodConcurrency 1Concurrency 2Concurrency 4Concurrency 8Concurrency 16
HumanEvalDFlash-3L649.061182.282013.902813.133290.63
HumanEvalDFlash-4L649.931193.382057.182760.293240.81
HumanEvalDFlash-5L641.931206.112037.602714.723204.59
HumanEvalLatentSpec648.491226.892070.462831.893342.65
MBPPDFlash-3L560.521081.231928.872681.793140.63
MBPPDFlash-4L567.041088.041923.822656.773142.50
MBPPDFlash-5L575.091105.681896.212610.023112.55
MBPPLatentSpec595.351113.631955.632740.423228.01

References

[1] EAGLE-4 Team. EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware. Preprint, 2026.

Q3: What are the measured latency and throughput of DSpark+LatentSpec relative to DSpark? Do the gains persist with generation limits of 2,048 or 4,096 tokens, and what are the actual output lengths and truncation rates?

A3. Table 3 reports the measured draft latency of DSpark and DSpark+LatentSpec. Compared with DSpark, DSpark+LatentSpec reduces draft latency by 2.05%–10.72% across context lengths of 512–16384. The gains from the lower FLOPs of DSpark+LatentSpec increase with context length.

Table 3: Draft latency (ms) across six context lengths under matched CUDA Graph conditions.

Context lengthDSpark (ms)DSpark+LatentSpec (ms)Reduction
5122.634 ms2.580 ms2.05%
10242.903 ms2.818 ms2.93%
20483.397 ms3.261 ms4.00%
40964.387 ms4.133 ms5.79%
81926.839 ms6.291 ms8.01%
1638411.656 ms10.407 ms10.72%

As shown in Tables 4 and 5, we measured the end-to-end throughput of DSpark and DSpark+LatentSpec across different workloads (mathematics, code, and chat), concurrency levels, and fixed output lengths. All results are averages over five runs. DSpark+LatentSpec outperforms DSpark in most configurations, with an overall end-to-end throughput improvement of 7.54%.

Using Qwen3-4B with a temperature of 1 and thinking mode disabled, we measured mean natural output lengths of 918 and 2999 tokens on MATH-500 and AIME25, respectively. Under the forced fixed output length of 512 tokens, the corresponding truncation rates are 55.80% and 100.00%.

Table 4: End-to-end aggregate throughput (tokens/s) on Code and Chat with a fixed output length of 512 tokens.

CategoryDatasetMethodConcurrency 1Concurrency 2Concurrency 4Concurrency 8Concurrency 16
CodeHumanEvalDSpark660.961238.752110.142881.703325.02
CodeHumanEvalDSpark+LatentSpec667.131249.392158.512987.793586.44
CodeMBPPDSpark610.211137.171918.802721.063202.51
CodeMBPPDSpark+LatentSpec619.991162.852036.732819.523434.70
ChatMT-BenchDSpark476.64899.561516.912147.692530.37
ChatMT-BenchDSpark+LatentSpec500.83885.151578.072243.562515.54
ChatAlpacaDSpark457.44854.511455.642066.802478.81
ChatAlpacaDSpark+LatentSpec464.76880.271527.242183.952619.77

Table 5: End-to-end aggregate throughput (tokens/s) on MATH-500 and AIME25 with fixed output lengths of 2048 and 4096 tokens.

DatasetMethodConcurrency 1Concurrency 2Concurrency 4Concurrency 8Concurrency 16
Output length = 2048 tokens
MATH500DSpark742.441362.212135.722796.072979.60
MATH500DSpark+LatentSpec763.181391.922301.183086.223368.13
AIME25DSpark659.271161.661751.032294.972563.80
AIME25DSpark+LatentSpec680.701207.341904.622646.722805.35
Output length = 4096 tokens
MATH500DSpark689.711227.761763.792207.562184.63
MATH500DSpark+LatentSpec719.291318.911967.932590.762556.53
AIME25DSpark598.551052.541523.351891.801864.13
AIME25DSpark+LatentSpec633.601096.991620.622242.052258.23

Distinct contribution of the latent mechanism

To investigate the contribution of the latent mechanism, we evaluated LatentSpec with only two latent tokens using Qwen3-4B as the target model. The results are shown in Table 6.

Table 6: Average acceptance length τ for Qwen3-4B under greedy decoding (T = 0) and sampling (T = 1).

Draft ModelMATH-500 (Math)AIME25 (Math)HumanEval (Code)MBPP (Code)MT-Bench (Chat)Alpaca (Chat)Overall Avg
Temperature T = 0 (greedy)
DFlash-4L5.274.714.464.192.842.714.03
LatentSpec-2_latent_tokens5.364.754.624.322.952.774.13
DFlash-5L5.364.734.594.332.962.774.12
LatentSpec5.384.774.624.352.992.784.15
Temperature T = 1 (sampling)
DFlash-4L4.563.734.123.892.662.563.59
LatentSpec-2_latent_tokens4.613.944.273.972.712.613.69
DFlash-5L4.623.904.253.942.722.593.67
LatentSpec4.673.974.343.992.742.593.72

The results show that LatentSpec with only two latent tokens (LatentSpec-2_latent_tokens) reduces FLOPs by 14.84%–16.62% relative to DFlash-5L at context lengths of 512–4096 while maintaining a similar average acceptance length. This indicates that, through a more effective allocation of computation, the Thinker can generate Latent Reasoning Hidden States that benefit the Drafter at a lower computational cost, yielding FLOPs savings during both training and inference.

If you have any further questions, we would be happy to answer them. If you find that our responses address your concerns, we would appreciate it if you could reconsider your score.

Thank you for the additional experiments. Though I am still not convinced by the latent reasoning interpretation, the new DSpark experiments do demonstrate some efficiency gains. I will therefore increase my soundness score and overall rating to 3

Reviewer_3J2sOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a Thinker-Drafter architecture for parallel speculative decoding. The Thinker reads cached target-model features and updates a small set of latent slots. The Drafter then uses this shared state to predict a candidate block in parallel, followed by standard target verification. On Qwen3-4B and Qwen3-8B, acceptance lengths are generally close to those of DFlash-5L. The paper reports lower drafting cost and higher end-to-end throughput in 18 of 20 evaluated settings on two math benchmarks.

Critical strengths
  1. The architecture offers a sensible way to reduce draft computation: two layers operate on four latent positions instead of all seven candidate positions. Table 1 shows that the saving translates into lower draft latency, with acceptance length largely preserved.

  2. The evaluation distinguishes draft latency from end-to-end throughput and tests several context lengths and concurrency levels. I also appreciate the discussion of DSpark in Section 5.4, where the authors explicitly acknowledge that the combined method brings little improvement.

Critical weaknesses
  1. I am not yet convinced by the "latent reasoning" interpretation. Section 5.3 adds two Thinker layers to a three-layer DFlash drafter, so it also adds capacity. Perceiver provides a useful architectural comparison: it concentrates computation in a small latent space for efficient feature processing. A DFlash control matched for FLOPs or latency would help distinguish the benefit of the proposed Thinker from a more general change in compute allocation.

  2. The additional acceptance gain over DFlash-5L is small: 4.12 to 4.15 on Qwen3-4B under greedy decoding, with a slight decrease on Qwen3-8B under sampling. The main benefit is therefore efficiency, but Table 3 only measures throughput on AIME25 and MATH-500 and gives no variability estimates. More evidence is needed to judge whether the smaller speed gains persist across runs and workloads.

Questions for authors
  1. Is there an existing comparison with a DFlash variant adjusted to similar FLOPs or draft latency? How were the four latent slots and the two/three-layer split selected?

  2. Were the throughput measurements repeated, and what was the variation? Are corresponding results available for code or dialogue workloads?

3 comments

Thank you for taking the time to review our work and provide valuable feedback. We address the comments below.

Q1. Is there an existing comparison with a DFlash variant adjusted to similar FLOPs or draft latency? How were the four latent slots and the two/three-layer split selected?

A1. We added a DFlash-4L draft model configuration. At context lengths of 512–4096, LatentSpec reduces FLOPs by 8.26%–9.41% relative to DFlash-5L, while DFlash-4L reduces FLOPs by 11.32%–12.45%. Reducing the number of layers from five to four does not reduce total computation by 20%: as discussed in EAGLE-4 [1], the LM head accounts for a large proportion of the draft model’s computation. We therefore selected DFlash-4L as a DFlash variant with FLOPs close to those of LatentSpec, and compared the average acceptance lengths of DFlash-4L, LatentSpec, and DFlash-5L (in ascending order of FLOPs) with Qwen3-4B as the target model.

Table 1: Average acceptance length τ for Qwen3-4B under greedy decoding (T = 0) and sampling (T = 1).

Draft ModelMATH-500 (Math)AIME25 (Math)HumanEval (Code)MBPP (Code)MT-Bench (Chat)Alpaca (Chat)Overall Avg
Temperature T = 0 (greedy)
DFlash-4L5.274.714.464.192.842.714.03
LatentSpec-2_latent_tokens5.364.754.624.322.952.774.13
DFlash-5L5.364.734.594.332.962.774.12
LatentSpec5.384.774.624.352.992.784.15
Temperature T = 1 (sampling)
DFlash-4L4.563.734.123.892.662.563.59
LatentSpec-2_latent_tokens4.613.944.273.972.712.613.69
DFlash-5L4.623.904.253.942.722.593.67
LatentSpec4.673.974.343.992.742.593.72

As shown in Table 1, LatentSpec achieves a higher overall average acceptance length than both DFlash-5L, which requires more computation, and DFlash-4L, which has a similar computational cost.

To examine the choice of four latent slots, we evaluated a draft model with two latent slots, as shown in Table 1. The results indicate that LatentSpec with only two latent tokens maintains an average acceptance length close to that of DFlash while reducing FLOPs by 14.84%–16.62% at context lengths of 512–4096. As Table 1 shows, two latent tokens are sufficient to maintain an average acceptance length close to that of DFlash-5L, while four latent tokens offer a trade-off between average acceptance length and FLOPs.

For the split of two Thinker layers and three Drafter layers, we considered the optimal stopping problem in probability theory and game theory, specifically the 37% rule. In our architecture, we allocate the first 2/5 of the layers, a proportion close to 37%, to observation and reasoning, namely generating Latent Reasoning Hidden States, and the remaining 3/5 to decision-making, namely draft generation. The experimental results in the paper and Table 1 demonstrate that this split effectively improves average acceptance length and end-to-end throughput.

References

[1] EAGLE-4 Team. EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware. Preprint, 2026.

Q2. Were the throughput measurements repeated, and what was the variation? Are corresponding results available for code or dialogue workloads?

A2. Regarding repeated measurements and variability, both the newly added experimental results and those reported in the paper are averages over five repeated runs. We have added the results for code and chat workloads in Table 2.

Table 2: End-to-end aggregate throughput (tokens/s) on Code (HumanEval, MBPP) and Chat (MT-Bench, Alpaca) with a fixed output length of 512 tokens.

DatasetMethodConcurrency 1Concurrency 2Concurrency 4Concurrency 8Concurrency 16
HumanEvalDFlash641.931206.112037.602714.723204.59
HumanEvalLatentSpec648.491226.892070.462831.893342.65
MBPPDFlash575.091105.681896.212610.023112.55
MBPPLatentSpec595.351113.631955.632740.423228.01
MT-BenchDFlash465.82880.741450.782087.162403.39
MT-BenchLatentSpec479.00885.291565.572171.162466.90
AlpacaDFlash447.63852.331450.862062.052423.60
AlpacaLatentSpec448.61843.931441.332065.002474.09

As shown in Table 2, together with the results on mathematical workloads in the paper, LatentSpec achieves higher throughput than DFlash in most cases on SGLang across mathematical, code, and chat workloads. This is enabled by its lower training and inference FLOPs and the allocation of computation between the Thinker and Drafter. The maximum throughput improvements on code and chat workloads are 5.00% and 7.91%, respectively. At high concurrency, the decoding stage becomes compute-bound, making LatentSpec’s advantage over DFlash more pronounced due to its lower FLOPs.

We welcome any further questions and would be glad to clarify any remaining points. If our responses have addressed your concerns, we would sincerely appreciate it if you could reconsider your rating.

Thanks for additional results. The compute controlled comparison and the code/chat dataset results already address my main concerns. Thus, I'm raising my score to weak accept.

Reviewer_tEspOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentSpec for parallel drafting in speculative decoding, building on methods such as DFlash. Instead of processing all draft positions through the same stack of layers, it splits the draft model into two stages. The Thinker first processes fewer latent slots and passes their states to the Drafter, which predicts the full draft block in parallel. Doing part of the computation on fewer positions improves efficiency, while the reported acceptance lengths remain close to those of DFlash.

Critical strengths

The paper studies a practical efficiency problem with a simple architecture change. I appreciate that it includes a five-layer DFlash baseline, matching the total depth of the Thinker and Drafter. It also reports actual draft latency and end-to-end throughput, in addition to FLOPs and acceptance length. These results provide evidence that the design can save computation and improve speed in the evaluated settings.

Critical weaknesses

My main concern is why this change can reduce computation without hurting draft quality. The paper suggests that the shared state helps coordinate the block, but does not show what this state contains or how different draft positions use it. The results show that the selected configuration works, but provide little evidence for the proposed explanation. Without such analysis, the method looks more like an architectural adjustment than a new latent-reasoning mechanism. The comparison with DFlash-3L adds two layers of computation, while the depth-matched five-layer comparison does not isolate the role of the shared state. Tests with different numbers of slots, different layer splits, or a simpler shared-feature module would help explain what is important about this design.

The improvements are also small, particularly in acceptance length, and no repeated-run variability is reported. It is therefore unclear how stable the smaller gains are. More results across training budgets or draft lengths would help show whether the improvement extends beyond the chosen configuration.

Questions for authors

Can the authors provide a controlled analysis explaining why the shared state preserves draft quality when part of the computation is moved to fewer slots, for example by varying the slot count or comparing with a simpler shared-feature module at a similar computation budget?

1 comment

Thank you for the thoughtful comments and helpful suggestions. Our responses are provided below.

Q1. Can the authors provide a controlled analysis explaining why the shared state preserves draft quality when part of the computation is moved to fewer slots, for example by varying the slot count or comparing with a simpler shared-feature module at a similar computation budget?

A1. To investigate whether the shared state can preserve draft quality when part of the computation is moved to fewer slots, we evaluated a draft model configuration with two latent tokens. Compared with DFlash, LatentSpec-2_latent_tokens reduces FLOPs by 14.84%–16.62% at context lengths of 512–4096.

For a comparison under similar computational budgets, we evaluated DFlash-4L. Reducing the number of layers from five to four does not reduce total computation by 20%. As discussed in EAGLE-4 [1], the LM head accounts for a large proportion of the draft model’s computation. Our calculations show that DFlash-4L reduces FLOPs by 11.32%–12.45% relative to DFlash-5L at context lengths of 512–4096. The FLOPs of DFlash-4L are close to those of LatentSpec, which reduces FLOPs by 8.26%–9.41% relative to DFlash. We therefore selected DFlash-4L as a baseline with a similar computational budget for comparison with LatentSpec.

Regarding variability in the experimental results, both the newly added results and those reported in the paper are averages over five repeated runs.

Table 1: Average acceptance length τ for Qwen3-4B under greedy decoding (T = 0) and sampling (T = 1).

Draft ModelMATH-500 (Math)AIME25 (Math)HumanEval (Code)MBPP (Code)MT-Bench (Chat)Alpaca (Chat)Overall Avg
Temperature T = 0 (greedy)
DFlash-4L5.274.714.464.192.842.714.03
LatentSpec-2_latent_tokens5.364.754.624.322.952.774.13
DFlash-5L5.364.734.594.332.962.774.12
LatentSpec5.384.774.624.352.992.784.15
Temperature T = 1 (sampling)
DFlash-4L4.563.734.123.892.662.563.59
LatentSpec-2_latent_tokens4.613.944.273.972.712.613.69
DFlash-5L4.623.904.253.942.722.593.67
LatentSpec4.673.974.343.992.742.593.72

As shown in Table 1, even with only two latent tokens and lower computation, LatentSpec achieves an average acceptance length close to that of DFlash. This indicates that the shared latent reasoning state generated by the Thinker has a positive effect on the Drafter’s generation. With an effective Thinker architecture and a suitable allocation of computation, draft quality can be maintained while benefiting from lower computational costs during both training and inference.

Under similar computational budgets, LatentSpec outperforms DFlash-4L across all evaluated datasets and sampling temperature settings. These results indicate that the Thinker in LatentSpec, through an architecture that allocates computation more effectively, generates a shared latent reasoning state that helps the Drafter coordinate the generation of the entire draft block.

We would be happy to provide any further clarification. If our responses have addressed your concerns, we would be grateful if you could reconsider your score.

References

[1] EAGLE-4 Team. EAGLE-4: One Flexible Drafter for All Batch Sizes and Hardware. Preprint, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.