acceptodds
Withdrawn from ICLR 2027

OneiroDraft: Reading the Latent Dreams for Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding accelerates autoregressive generation by pairing a lightweight drafter with parallel verification by the target model. Its effectiveness depends on producing accurate candidate tokens at sufficiently low drafting cost. Existing approaches make different compromises between these objectives: independent draft models require separate model computation and may diverge from the target model, while auxiliary prediction heads remain closely coupled to the target but are typically most effective over short prediction horizons. We explore an alternative based on the target model’s intermediate hidden states, which already encode rich contextual information and can be extrapolated through lightweight transitions instead of repeated full-model computation. We introduce OneiroDraft, a recurrent latent drafter that adapts a small contiguous interval of the pretrained target Transformer into a shared transition for predicting future hidden states and converting them into candidate tokens. To improve prediction over multiple steps, we first train the transition and readout modules separately and then jointly optimize them over recurrent rollouts, aligning state prediction with token generation. During inference, OneiroDraft recurrently generates drafts with the learned transition and combines them with contextual candidate reuse to reduce drafting overhead, while a configurable verification criterion controls the trade-off between strict target matching and candidate acceptance. On Qwen3-8B, recurrent latent drafting alone achieves 1.20×–1.27× wall-clock speedup, while the Hybrid decoder that combines it with contextual reuse reaches 1.59×–1.70× wall-clock speedup and 1.60×–1.70× throughput improvement over paired greedy decoding. These results show that pretrained target computation can be repurposed into an inexpensive recurrent proposal mechanism: the adapted target interval advances hidden representations along the target model’s future trajectory and produces useful speculative candidates without repeatedly executing the complete target model.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative decoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HpeFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

OneiroDraft trains a transition module copied from target-model layers to predict hidden states at subsequent token positions. A readout converts these predictions into draft tokens, while recurrent calibration improves multi-step prediction. A hybrid route uses n-gram lookup when possible and neural drafting otherwise. Evaluation on Qwen3-4B/8B and mathematics benchmarks reports 1.20–1.27× speedups for neural drafting and 1.59–1.70× for the hybrid system over paired greedy decoding. The paper also studies numerical differences between speculative verification and serial greedy execution.

Critical strengths
  1. The calibration experiments provide useful controls. Joint and Extended calibration start from the same Staged checkpoint with a fixed target, helping isolate recurrent calibration. On Qwen3-8B MATH-500, latent acceptance rises from 19.44% to 26.68% and Latent-only speedup from 1.16× to 1.27×. The initialization and training-budget comparisons also expose limitations rather than reporting only favorable results.
Critical weaknesses
  1. The learned drafter's added value is not established against existing methods. The evaluation compares with greedy decoding and the authors' lookup route, but includes no trained or self-speculative drafter. Most hybrid gains come from lookup: the neural route adds only 0.06–0.12× on Qwen3-8B, while Hybrid matches or underperforms Lookup-only across all seven Qwen3-4B benchmarks after Joint calibration. These results make the benefit of training the transition module uncertain. A matched comparison with a strong learned drafter is more informative than comparing speedups reported under different experimental settings.

  2. The evaluation also leaves the acceleration and quality claims difficult to interpret. The target is adapted with LoRA, so results against the same checkpoint do not establish performance relative to unmodified Qwen3. Appendix B.3 states that GPU models were not consistently recorded, and hardware matching within timing pairs is unclear. The main 1024-token cap substantially affects reasoning accuracy. Only about 23–26% of outputs exactly match serial greedy, and the longer-generation results include accuracy decreases. The diagnosis of one numerical divergence does not by itself establish quality preservation across the evaluation. Controlled hardware, a clearly specified target, and longer-generation quality measurements are needed.

Questions for authors
  1. Is target LoRA enabled and merged during both greedy and speculative inference? How do the adapted and unmodified targets compare in accuracy and greedy latency under the same protocol?

  2. Were paired runs performed on the same GPU, and which GPU models were used? Was thinking mode enabled, and what fraction of responses reached the generation cap?

  3. Does the neural route provide a meaningful advantage over a strong learned or self-speculative drafter with matched hardware, target, generation length, and lookup routing? Please also report training cost and quality over longer generations.

Reviewer_2DGiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

OneiroDraft drafts with the target model's own computation. Two adjacent layers of the target Transformer are copied, given LoRA, and re-applied across token positions as a shared transition from a mid-stack hidden state; a boundary adapter reads tokens from the predicted states. Training fits the transition on teacher-forced states, then calibrates it jointly over undetached rollouts. Each round commits the target's own anchor plus the accepted prefix, and inference routes to exact-match context lookup whenever a continuation exists, which the paper calls Hybrid. Against paired greedy decoding as the only baseline, on Qwen3-8B and seven mathematical-reasoning benchmarks, the Hybrid route reaches 1.59-1.70x wall-clock and the neural route alone 1.20-1.27x. An appendix audit reports that 23-26% of speculative runs reproduce their paired greedy sequence.

Critical strengths
  1. The three-arm decomposition is the right instrument, and it is published in full, including the arm that does not favour the method. Tables 3, 12 and 13 report latent-only, lookup-only and hybrid side by side across three training recipes and two model sizes, so a reader can see where each route's cost actually goes instead of only the composite. Appendix A.6 then turns the design into a checkable budget: writing the round yield as G_K and requiring C_D + C_V < G_K C_A states precisely when recurrent drafting can pay for itself, which is the kind of accounting that lets a reader locate the ceiling of the method rather than accept an end-to-end number.

  2. Appendix D is an unusually good audit, and self-critical numbers are reported where they hurt. The paper instruments a divergent position, contrasts a 34.50 versus 34.25 greedy margin with a 34.25 versus 34.25 block tie, traces the first difference to a first-layer query projection at 3.91e-3, and reports that per-token linear layers restore element-wise identical logits while a local FP32 control restores the same top-1; Sections 4.1 and D say plainly that strict verification is not sequence equivalence to serial greedy. Alongside that, Appendix A.3 states that the identity map beats the copied transition on all three geometric metrics and that token-level metrics were not computed for it, Table 11 states that the two regimes do not share a target LoRA trajectory and therefore compare recipes rather than isolate a cause, and Appendix C.5 states which benchmarks lack three-seed coverage.

Critical weaknesses
  1. The only baseline is paired greedy decoding, and it is not a fixed model. EAGLE, Medusa, Hydra and ReDrafter appear as qualitative rows in Table 1 and nowhere as measurements; DFlash, Lookahead and n-gram lookup are not mentioned at all. Speedups are relative to a serial baseline in raw PyTorch with SDPA at batch one, no serving framework and no concurrency, which is the slowest autoregressive configuration available. Trainable target LoRA is also applied, so the paired model moves with the recipe: greedy accuracy on AMC 2023 is 27.50 under extended calibration and 35.83 under full-joint. No published drafter is measured anywhere.

  2. The training-free arm supplies most of the speedup. Table 3 puts latent-only at 1.20-1.27x, lookup-only at 1.50-1.64x and hybrid at 1.59-1.70x, so the entire neural machinery adds 0.06-0.12x, a mean of 0.089. On Qwen3-4B, Table 12 puts hybrid below lookup-only in thirteen of fourteen cells and level in the fourteenth, never above. Section 4.3 states that combining the two yields the highest speedup on every benchmark: that holds for Table 3, is contradicted by Table 12, and the reversal is never discussed in the text.

  3. The reported range sits at the arithmetic ceiling of a two-token horizon. With two neural drafts per round and consecutive acceptance, a round yields 1 + p + p^2 tokens. At the reported 26.68% latent acceptance that is 1.34, and the reported 1.27x is 95% of it; at the staged 19.44% it is 1.23 and 1.16x is 94%. Essentially no headroom remains except a longer draft, which is not attempted and which Section 5 mentions only as a general trade-off. Neither the abstract nor Section 4.3 says this, so 1.20-1.27x reads as a quality result rather than a horizon limit.

  4. The method is not lossless, and the paper does not say what the lossless configuration costs. Table 18 reports exact agreement with the paired greedy sequence of 23.15% to 26.46%. Sections 4.1, B.3 and D state this plainly and diagnose one case as a BF16 kernel-shape difference that turns a 34.50 versus 34.25 margin into a 34.25 versus 34.25 tie. They also report that per-token linear layers restore element-wise identical logits and that a local FP32 control restores the same top-1, yet neither configuration is timed or aggregated, so the configuration that would make the headline numbers comparable is described but unreported.

  5. The configurable verification criterion is lossy by construction and is not described that way. Appendix B.3 defines it as accepting any candidate whose relative target support reaches lambda, so at lambda = 0.90 the committed token may differ from the target argmax, and no rejection-sampling correction follows. Section 4.5 discusses the threshold only in wall-clock terms, while Table 16 shows lambda = 0.90 moving accuracy by up to 2.50 points on AMC 2023 and 2.23 on AIME 2024. The abstract presents the knob as a trade-off between strict matching and acceptance, which reads as free, and its cost is never measured.

  6. The timing provenance cannot carry a wall-clock claim. Appendix B.3 states that runs were distributed across remote GPU workers and that the accelerator model was not recorded per run, so the results characterise a mixed-worker setting rather than one device. Pairing is defined on the checkpoint alone and nothing says the two runs of a pair executed on the same device, so a ratio can absorb hardware heterogeneity. Absolute throughput, GPU hours, parameter counts, training tokens, lookup hit rate and mean accepted length are all absent, and inference is batch one with no concurrency and no serving framework.

  7. The single cell that separates OneiroDraft from every row of Table 1 is the copied target block, and the probe meant to support it does not. Table 6 has the identity map, which reuses h_t unchanged, beating the copied transition on all three geometric metrics: 0.553 against 0.654 hidden MSE, 3.015 against 3.602 relative MSE, and 0.542 against 0.436 cosine. Token-level metrics were not computed for identity, and the only large margin is against base weights resampled from N(0, 0.02). No comparison is made against a small trained transition of matched cost.

Revisions needed: add at least two published drafters in the same stack, measured both at batch one and in a batched serving setting, so the headline number has a reference point; report the exact-agreement rate from Table 18 alongside the speedup of the lossless verification configuration and state the gap between the two; report the lookup hit rate, the mean number of proposals and acceptances per round, and a latency breakdown between drafting and verification; sweep the draft horizon after calibration and report where the round-yield bound binds rather than treating the two-token horizon as given; and state the two-token horizon and the token-yield bound explicitly in Section 4, describe lambda < 1 as a lossy setting with its distributional consequence, and separate the target-LoRA effect from the drafter effect.

Questions for authors
  1. Which published drafter do you expect OneiroDraft to beat on Qwen3-8B at the same batch size, and what are the absolute tokens per second for greedy and for hybrid?

  2. On Qwen3-4B, Table 12 places hybrid below lookup-only in thirteen of fourteen cells. Under what reading is the neural route additive at that scale, and why is this reversal absent from the text?

  3. With two drafts per round and 26.7% acceptance a round yields 1.34 tokens, which accounts for the reported 1.27x to within 5%. What did you measure at K = 4 or K = 8 after calibration, and what bounds the horizon?

  4. What is the wall-clock speedup under the per-token linear-layer or local FP32 configuration that restores element-wise identical logits, and what happens to output length and distribution at lambda = 0.90?

  5. Were the greedy and speculative runs of each pair executed on the same device in the same session? Which accelerator was used, and how much does the ratio move between workers?

Reviewer_jpiWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OneiroDraft, a recurrent latent-state drafter for speculative decoding. It reuses and fine-tunes a block of the target model's Transformer layers to recursively predict future hidden states, which are then converted into draft tokens. The authors also combine it with contextual reuse to improve decoding efficiency. Experiments on Qwen3-4B and Qwen3-8B across several math benchmarks show speedups over standard greedy decoding.

Critical strengths

None

Critical weaknesses
  1. Limited novelty. Existing speculative decoding methods also fine-tune draft models to recursively predict future tokens. OneiroDraft primarily differs by initializing its recurrent drafter from copied target-model Transformer layers, but it remains unclear whether this design offers a fundamental advantage over existing draft models.

  2. Limited evaluation. The experiments cover a limited set of models and datasets, making it difficult to assess the generality of the proposed method across different model architectures and tasks.

  3. Insufficient differentiation from prior work. The overall framework follows the conventional speculative decoding paradigm of generating draft tokens and verifying them with the target model. The paper does not sufficiently clarify the conceptual or algorithmic novelty beyond its recurrent latent-state drafting mechanism.

Questions for authors
  1. How does OneiroDraft compare with existing recurrent draft models under the same model size, training budget, and inference settings? Does reusing target-model layers provide a clear advantage?

  2. Can the authors clarify the key technical novelty of OneiroDraft beyond the standard speculative decoding pipeline and recurrent latent-state prediction?

  3. How well does OneiroDraft generalize to other model architectures and non-mathematical tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.