acceptodds
Under review as a conference paper at ICLR 2027

IMLE-LLM: Latent-Conditioned Windowed Decoding for Faster Inference

Abstract

Autoregressive language models emit one token per forward pass, so generation latency grows linearly with output length. Writing several tokens per pass would cut this latency, but tokens written in the same pass cannot see one another. Most existing methods therefore recover the missing dependence outside the pass, with a draft model and a verifier or with extra passes. Yet tokens of one pass can all see the same random draw. We propose IMLE-LLM, a fine-tuning recipe that can turn any pretrained decoder-only transformer into a windowed decoder, adding only a small latent projection. All positions of a window share one Gaussian latent, so a single forward pass commits a window of mutually dependent tokens, with no draft model, verifier or extra passes. We fit this implicit model with conditional implicit maximum likelihood estimation (conditional IMLE), which needs no network that infers the latent. Built on Qwen3-8B, IMLE-LLM reaches a mean speed-up of 3.39× over one-token decoding while keeping most of the accuracy of an autoregressive fine-tune.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-token decoding, parallel decoding, latent-variable models, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qmwpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper, "IMLE-LLM: Latent-Conditioned Windowed Decoding for Faster Inference," introduces a post-hoc method to improve the inference speed of LLMs. The key idea of this study is to add a latent matrix that connects tokens and draws windows to accelerate decoding. The authors implement the proposed method on Qwen3-8B and have shown 3.44 times acceleration on the W=4 pattern, along with a mild degradation of performance by a few percent from the original Qwen3-8B. The authors also provide ablation studies regarding training hyperparameters and window size.

Critical strengths

I can see this paper has a chance to be a good paper for the following reasons:

  1. The idea of adding a latent matrix to an existing LLM to connect tokens is very impressive. Generally, these kinds of work requires from scrach training, but the proposed method does not require it.
  2. I think the performance that achieved by the proposed method is marginally above the acceptance criteria, even if it does not achieve state-of-the-art.
Critical weaknesses

However, I see several critical weaknesses that prevent me from giving acceptance scores:

  1. The authors cite many related papers, but the comparison to existing methods is too poor. The number of tables in Table 1 seems to be just a pick-up of existing papers, not a reproduction, and some of them are not Qwen3-8B. As the authors state in the related work and introduction, this area is dense, and there are many methods should be compared with the proposed method.
  2. The method has only been verified in Qwen3-8B; I think it is not difficult to implement the proposed method on another major LLM such as Gemma or Llama. Two LLM architectures should be a minimum criterion.
  3. The authors have validated the proposed method in non-reasoning mode; however, the accuracy degradation of reasoning mode on such method is very important and generally hindered. The authors should try the reasoning mode.
  4. I can see small statement regarding the compute cost of fine-tuning; however, this is not enough. The authors should conduct experiments to study the computing cost.
  5. The authors should also analyze the memory cost.
  6. The description of the fine-tuning in the main text is too abstract to understand the exact method. The appendix is very good for understanding the details. The authors should add some figures and statements to main text from appendix.
Questions for authors

The authors should add additional analysis: existing methods, different architectures, memory cost, and fine-tuining cost. My most important concern is about reasoning mode performance. Regarding details, see weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qmwp

Thank you for your review.

W3. "My most important concern is about reasoning mode performance"

W = 4 works in reasoning mode. We retrained it on Qwen3-8B's own thinking answers, with answers of up to 32,768 tokens. Over six tasks its accuracy is 94.3% of the original Qwen3-8B's, at 3.46× at batch 1 and 3.38× at batch 4 (Table G4 in the general response, Table 3 in Section 4.2). On MATH-500 and LiveCodeBench it gets 97.73% and 93.40% of the original, vs. 95.14% and 90.63% without thinking (Appendix B.3).

Speed on long answers is close to the speed without thinking (3.39× and 3.28× at batch 1 and 4). W = 4 has "no draft model and no verifier" (L291-294 of the submission). Long reasoning has no draft acceptance rate to pull down.

W1. "the comparison to existing methods is too poor"

We added EAGLE-3, I-DLM, Jacobi Forcing and K-Forcing to the comparison. Among the lossy decoders, W = 4 is the fastest at 3.39× and has the highest code retention, 92.93% (Tables G1 and G3, Table 2 in Section 4.2 and Table 11 in Appendix F). WeDLM, with its threshold raised to the 3.88 tokens per pass of W = 4, loses 13.91 points on the six-task mean at 3.19×, while W = 4 loses 5.16 points at 3.39× (Appendix F).

W1 (continued). "just a pick-up of existing papers, not a reproduction, and some of them are not Qwen3-8B"

Table 1 is our own measurement. Every speed cell is our timing (Appendix D.2), and every accuracy cell is our run of the released code and weights under one protocol and one grader (Section 4.1 of the revised PDF, Appendix C.1). Five cells, now in blue, had come from the baselines' papers, which score with other protocols (Kirchenbauer et al. score GSM8K few-shot with flexible answer extraction). The reruns change no speed-up. As an outside check, our 43.07 IFEval for the Magpie SFT base is within a single prompt of the Open LLM Leaderboard's independent strict score of 42.88 for the same model.

Nemotron-LD and Kirchenbauer et al. use other 8B bases, so Table 1 compares each with its own base. Kirchenbauer et al. get 91.28% of their base's math and 84.43% of its code at 2.41×, while W = 4 gets 95.65% and 92.93% of Qwen3-8B's at 3.39×.

W2. "Two LLM architectures should be a minimum criterion"

The submitted recipe and settings work on Llama-3.1-8B-Instruct, one of the two models the review suggests. W = 4 runs at 3.38× at batch 1 and gets 96.51% of math and 92.02% of code of the original (Table G5, Tables 14 and 15 in Appendix G.3). Llama has no QK-norm and 32 layers, not Qwen3-8B's 36. Its speed-up is still near Qwen3-8B's 3.39×.

W4 and W5. "study the computing cost" and "analyze the memory cost"

Fine-tuning W = 4 costs little more than DFlash's own drafter recipe, and serving it needs almost no extra memory. Fine-tuning takes 699.27 H100-hours and DFlash's recipe 632.46, each timed on one node of 8 H100 (Table G6, Table 5 in Appendix B.2). With BAdam, which trains one layer at a time, the W = 4 fine-tune peaks at 28.0 GiB per card, where full AdamW needs about 137.3 GiB (Appendix B.2). At serving, W = 4 adds 0.1 GiB to the original's 15.9 GiB at batch 1, and DFlash adds 2.0 GiB (Table G6, Table 10 in Appendix E.2).

W6. "The description of the fine-tuning in the main text is too abstract"

Section 3.2 now gives the five steps of one training round of Algorithm 1, and Figure 5 in Appendix A.1 shows the windows and the weight of each token.

  1. Cut a group of 48 examples into windows of W tokens at stride 1.
  2. With probability ρ = 0.25, the current model rewrites one window per example.
  3. Each window gets m = 4W latent draws, and the lowest-loss draw is used.
  4. Take 12 update steps of 8 examples each on Equation 5.
  5. In each step, train Wz\mathbf{W}_z and one transformer layer, as BAdam does.
Reviewer_qmwpOpenReview ↗

Reply from reviewer

Dear Authors

Thank you for the detailed rebuttal and additional experiments. I appreciate the improvements, particularly the thinking-mode evaluation and expanded baseline comparisons.

However, the results remain mixed regarding thinking mode. While IMLE-LLM achieves a 3.46× speedup in thinking mode, it shows noticeable accuracy degradation on AIME 2024 and AIME 2025. Moreover, DFlash with the proposed IMLE objective achieves a higher speedup of 4.33× without sacrificing accuracy.

To clarify the practical advantages of the proposed method, I request two additional experiments:

  1. Per-benchmark speed–accuracy comparison against DFlash and DFlash with IMLE under matched thinking-mode settings.
  2. Accuracy versus reasoning length, to examine whether errors accumulate over longer reasoning sequences.

Although the revision substantially strengthens the paper, I remain unconvinced that the speed–accuracy trade-off is sufficiently competitive for challenging reasoning tasks. I therefore maintain my rating of Weak Rejection.

Best

AuthorsOpenReview ↗

Response to Reviewer qmwp, follow-up

Thank you for the follow-up. Both experiments are below and will go into the appendix.

"DFlash with the proposed IMLE objective achieves a higher speedup of 4.33× without sacrificing accuracy"

The 4.33× comes from the proposed objective. W = 4 is trained with it as one model, and DFlash's drafter is trained with it while Qwen3-8B checks every draft. With thinking, the drafter trained with our objective is faster than with DFlash's own loss on all six benchmarks, by 1.13× to 1.39× at block 16, and most on AIME. At block 32 this drafter is also faster than W = 4 on math and code, so we recommend it for exact output on hard reasoning. The final version will say in the abstract which decoder suits which use.

"Per-benchmark speed-accuracy comparison against DFlash and DFlash with IMLE under matched thinking-mode settings"

All rows use the Table G4 setting, and each speed-up is the mean of three timing runs that agree within 0.03×.

Table F1. Thinking mode on Qwen3-8B, per benchmark (speed-up at batch 1)

BenchmarkOriginal accuracyW = 4 accuracyW = 4DFlash, retrained, block 16DFlash with our objective, block 16DFlash with our objective, block 32
MATH-50097.094.83.45×2.92×3.80×5.40×
AIME 202475.069.03.48×2.71×3.77×5.22×
AIME 202566.560.03.47×2.66×3.70×5.12×
GPQA-Diamond61.659.63.48×2.24×2.93×3.30×
LiveCodeBench51.548.13.35×2.68×3.59×4.97×
IFEval84.780.83.53×1.73×1.95×1.97×
Mean, batch 13.46×2.49×3.29×4.33×

W = 4 is faster than the retrained DFlash on every benchmark, and faster than every drafter on GPQA-Diamond (3.48× against at most 3.30×) and IFEval (3.53× against at most 1.97×), where drafts are often rejected. It needs no drafter and adds 0.1 GiB of serving memory, where DFlash adds 2.0 GiB (Table G6, 4K answers).

"Accuracy versus reasoning length, to examine whether errors accumulate over longer reasoning sequences"

W = 4 keeps at least as large a share of the original's accuracy on long answers as on short ones. We sorted each benchmark's problems into thirds by the original's thinking length (without IFEval, whose thinking is short). Cells give original vs. W = 4 accuracy in %, with the kept share in parentheses.

Table F2. Accuracy versus reasoning length (thinking mode)

BenchmarkShort thirdMiddle thirdLong third
MATH-50099.4 vs. 97.1 (97.7%)97.8 vs. 95.7 (97.9%)93.8 vs. 91.6 (97.7%)
AIME 202492.5 vs. 84.4 (91.2%)77.5 vs. 71.3 (92.0%)55.0 vs. 51.3 (93.3%)
AIME 202588.1 vs. 79.4 (90.1%)68.8 vs. 61.9 (90.0%)42.5 vs. 38.8 (91.3%)
GPQA-Diamond70.5 vs. 68.0 (96.5%)61.9 vs. 60.0 (96.9%)52.5 vs. 50.8 (96.8%)
LiveCodeBench74.7 vs. 69.8 (93.4%)51.3 vs. 47.7 (93.0%)28.6 vs. 26.9 (94.1%)
AIME 2024 and 202590.3 vs. 81.9 (90.7%)73.1 vs. 66.6 (91.1%)48.8 vs. 45.0 (92.2%)
Pooled, five benchmarks85.2 vs. 81.4 (95.5%)73.3 vs. 70.3 (95.9%)59.7 vs. 57.6 (96.5%)

The AIME 2024 and 2025 row has 20 problems per third, and the five-benchmark row about 400, and its 95% intervals are at most ±1.5 points (paired bootstrap). Longer thinking also goes with harder problems, so across the five benchmarks we controlled for difficulty with the original's accuracy on each problem. The kept share then changes by +0.2 points per 10K tokens of thinking (95% interval −0.6 to 1.0). Within each AIME problem, where difficulty is fixed, we compared each model's 8 longest and 8 shortest samples. The original's longer samples are 17.3 points less accurate (62.1% against 79.4%), and W = 4's are 15.6 points less accurate (56.7% against 72.3%), so W = 4 keeps 91.1% and 91.3% in the two halves.

We also scored W = 4's AIME answers with Qwen3-8B. Under the speculative-sampling rule, it accepts 93.3% of their tokens in the first 4K tokens, and 92.9%, 93.2% and 93.0% from 4K to 8K, from 8K to 16K and beyond 16K. W = 4's AIME thinking is 1.03× as long as the original's, and 2.9% of them hit the 32,768-token budget, against 2.6%.

"noticeable accuracy degradation on AIME 2024 and AIME 2025" and "sufficiently competitive for challenging reasoning tasks"

W = 4 loses 6.0 points on AIME 2024 and 6.5 on AIME 2025, and the share it loses is about the same on the shortest and the longest problems (Table F2). When AIME answers must be exact, the drafter trained with our objective gives 5.22× and 5.12× at block 32 with no loss, and at block 16 it gives 3.77× and 3.70× against 2.71× and 2.66× for DFlash's own loss. With one model and no drafter, or on open-ended text, W = 4 is the faster choice.

The reviewer's requests have made the paper stronger, and we thank the reviewer for them. If these results answer the concern on reasoning, we would be grateful if the reviewer considered raising the score.

Reviewer_KT3rOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed IMLE-LLM, a fine-tuning recipe that can turn any pretrained decoder-only transformer into a windowed decoder, adding only a small latent projection. All positions of a window share one Gaussian latent, so a single forward pass commits a window of mutually dependent tokens, with no draft model, verifier or extra passes. Built on Qwen3-8B, IMLE-LLM reaches a mean speed-up of 3.39× over one-token decoding while keeping most of the accuracy of an autoregressive fine-tune.

Critical strengths
  1. This paper is very clearly written, providing detailed technical details on training and inference.

  2. The paper investigates a crucial problem: how to address the token-dependency disruption issue arising from parallel decoding.

  3. The proposed method is interesting.

Critical weaknesses
  1. The core idea of ​​the proposed method seems to share many similarities with K-Forcing [1]. The authors should discuss the main differences and advantages of the proposed method when compared to K-Forcing.

  2. While the proposed method is interesting, it's not practical. IMLE-LLM cannot achieve the high token-per-forward of a diffusion language model, nor can it achieve lossless acceleration like speculative decoding.

  3. The experimental section lacks comparisons with some important baselines, such as IDLM [2].

  4. Can the proposed method help improve draft quality in dflash?

  5. The speed–accuracy trade-off is not sufficiently competitive.

[1] K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling

[2] Introspective Diffusion Language Models

Questions for authors

Please see the weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer KT3r

Thank you for the review and for pointing us to K-Forcing and I-DLM. To make the method easier to follow, Section 3.2 now goes through one training round in five steps, from cutting a group of 48 examples into windows to the BAdam update (Figure 5 in Appendix A.1).

W1. "seems to share many similarities with K-Forcing"

Both feed noise to a block of future slots and decode it in one pass, and differ in the noise and how the model learns to use it. K-Forcing draws one uniform scalar per future token and embeds the scalars with an encoder of about 168M parameters at Qwen3-8B's width. It trains in three chained stages with a frozen teacher. Our W = 4 gives one Gaussian vector per window to every slot through 0.5M parameters and trains in a single stage at the final W with no teacher during training, a single-stage recipe that K-Forcing's paper lists as open (its Section 5). One shared latent is the better choice in our recipe, since a separate latent per slot scores lower. The per-slot model scores 0.61 to 1.85 points below ours on five tasks and is level on LiveCodeBench, at the same 3.39× (Appendix F.1).

Under the protocol of Table 1 on Qwen3-8B, the simpler design also scores higher and runs faster. At 3.39×, W = 4 gets 95.65% math retention, 92.93% code retention and 71.23 on IFEval. K-Forcing gets 88.06%, 82.56% and 59.52 at 2.85×. Ours runs 1.19 times as fast (Table G3, Table 11 in Appendix F).

K-Forcing's README lists the training recipe as not yet released, so we rebuilt its training loop. We first checked the rebuild in K-Forcing's own LM1B setting, where it gets a Gen-PPL of 129.9 at k = 4 (its paper reports 127.6), and then trained it on Qwen3-8B. We tried four teacher temperatures where the paper uses 1.0 and took the best validation checkpoint, with the better of its sampler's frequency penalty or none. Training compute matched our W = 4 run.

W2. IMLE-LLM is not practical

Our W = 4 commits 3.88 tokens per pass, more than the diffusion decoders it is compared with, WeDLM (3.31), ReFusion (2.65), SDAR (2.14) and I-DLM at stride 4 (2.69) (Table 1, Table G3). Where exact output is needed, our objective inside DFlash's drafter gives lossless output at 4.75× and 6.23× at batch 1 (W4 below). W = 4 itself is one fine-tuned model with 0.5M new parameters and no drafter. Its fine-tune takes 699.27 H100-hours vs. 632.46 for DFlash's drafter recipe. At batch 16 W = 4 runs at 2.86× vs. 2.67× for our from-scratch DFlash and 1.09× for EAGLE-3, adding 1.0 GiB where DFlash adds 3.2 GiB (Tables G1, G6).

W3. "lacks comparisons with some important baselines, such as IDLM"

Our W = 4 runs faster than I-DLM-8B at stride 4, with and without thinking. Using its released weights and code, our harness first reproduced its H100 speed-up (2.27× vs. 2.30×, Appendix F), which backs the 2.26× we measure for it under the protocol of Table 1, while W = 4 gets 3.39×. I-DLM keeps more math and IFEval (97.53% and 80.22 vs. 95.65% and 71.23), and W = 4 leads on code (92.93% vs. 91.08%) (Table G3, Table 11 in Appendix F). In thinking mode, where I-DLM is trained, the gap remains at 2.29× vs. 3.46× (Appendix B.3).

W4. "Can the proposed method help improve draft quality in dflash?"

Yes. Our drafter and DFlash's are both trained from scratch on DFlash's recipe. Ours adds one Gaussian latent per block through Wz\mathbf{W}_z. It accepts 5.86 tokens per check against 4.49 at block 16 and 7.98 against 4.61 at block 32, both greedy. Qwen3-8B checks every draft, so the output is Qwen3-8B's own, and the batch-1 speed-ups are 4.75× vs. 3.61× and 6.23× vs. 3.62× (Table G1, Table 2 in Section 4.2). With Wz\mathbf{W}_z but a single draw (m = 1), the block-16 drafter accepts only 4.56 (Appendix B.4). One draw is independent of the target block, so the drafter learns to ignore it, as in ablation C of the submission (L398-403).

W5. "The speed-accuracy trade-off is not sufficiently competitive"

W = 4 is the fastest lossy decoder in Table G3 at 3.39× and scores highest on code. It is more accurate than WeDLM, ReFusion and SDAR on all six tasks and faster than each (Table 1, Figure 8 in Appendix F). With WeDLM's threshold raised to our 3.88 tokens per pass, it still runs slower and loses more, 13.91 points on the six-task mean at 3.19×, while W = 4 loses 5.16 at 3.39× (Appendix F).

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for the time and effort spent reviewing our paper. As the discussion period ends soon, we would be grateful to know whether our response has addressed the reviewer's concerns. If it has, we kindly ask the reviewer to consider raising the score. If any concern remains, we would be happy to address it in the time left.

Thanks for the rebuttal. After reading the rebuttal, I would like to raise my score to 3.

AuthorsOpenReview ↗

Response to Reviewer KT3r, Thank you

Thank you for reading our rebuttal and for raising the score. We are grateful for your time and effort, and for the constructive feedback that improved our paper.

Reviewer_dzDpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper fine-tunes a pretrained LLM to write several tokens per forward pass without a draft model or verifier. Since tokens produced in the same pass cannot see each other, the authors feed all W slots of a window the same projection of one Gaussian latent, so the tokens become dependent through the shared draw. The model is trained with conditional IMLE, which updates only on the nearest of m latent draws and adds just a 0.5M-parameter projection. On Qwen3-8B with W = 4, IMLE-LLM reports a 3.39x mean speed-up at batch 1, keeps 97.3% of the math and 95.2% of the code accuracy of an autoregressive fine-tune, and beats three diffusion decoders on the same base in both accuracy and speed.

Critical strengths
  • Simple idea with careful evaluation. Sharing one latent across a window is a clean way to couple tokens within a single pass, the m = 1 ablation (a 9.35-point drop, Table 2) shows that multiple draws are needed, and Table 4 shows that different draws change 34.2% of windows at W = 4. The speed measurements are well controlled (Appendix E.2), and the method beats WeDLM, ReFusion and SDAR on the same base model in both accuracy and speed.
Critical weaknesses
  • IMLE-LLM loses accuracy and is still slower than an existing lossless baseline. DFlash, a lossless speculative decoder on the same Qwen3-8B and GPU, reaches a 3.64x mean speed-up against 3.39x for IMLE-LLM and is faster on five of the six tasks (Table 1). IMLE-LLM, in contrast, loses accuracy on all six generated benchmarks, and the one task where it is faster, IFEval, is also where it loses the most. The paper does not explain why one would choose a lossy decoder that is slower than an available lossless one, and it does not measure DFlash's memory, so the benefit of having no drafter is not shown.

  • The accuracy cost is larger than the retention numbers suggest. The 97.3% and 95.2% figures are relative to the AR fine-tune, which is already below the original Qwen3-8B on every generated benchmark, while the speed-up is measured against the original model. Against the original, IMLE-LLM drops 3.6 points on GSM8K, 5.7 on HumanEval and 10.8 on IFEval, and IFEval falls to 59.95 at W = 5. All results use one backbone in non-thinking mode, so the trade-off is not shown for long reasoning outputs, where decoding speed matters most.

Questions for authors
  • How does IMLE-LLM compare with DFlash and EAGLE-3 at batch 4 and 16, where Table 6 shows its speed-up over the same weights decoded one token at a time falling to 3.28x and 2.86x?
  • Can the window be used as a draft for a verifier, and how would its acceptance length compare with existing drafters?
  • Why is IFEval affected much more than the other benchmarks, and does the gap persist in thinking mode?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dzDp

Thank you for the detailed review.

W1 and Q2. "slower than an existing lossless baseline" and "Can the window be used as a draft for a verifier"

Yes, as a small drafter, and in that form it beats the lossless baseline. A drafter trained with our objective writes each window in one pass from one shared latent, and Qwen3-8B checks every draft, so the output is Qwen3-8B's own, as with any speculative decoder. At block 16 it accepts 5.86 tokens per check and runs at 4.75×, compared with 4.49 and 3.61× for DFlash's own loss. At block 32 it gets 7.98 and 6.23× vs. 4.61 and 3.62×. EAGLE-3 accepts 2.16 (Table G1 in the general response, Table 2 in Section 4.2). Both drafters of a block size are trained from scratch on DFlash's recipe and differ only in the objective and latent projection. With one draw (m = 1) the block-16 drafter accepts only 4.56, so the gain comes from the draws (Appendix B.4).

Without any drafter, W = 4 is faster on open-ended text. It runs at 3.54× on chat and 3.51× on IFEval, while the released DFlash is at 1.89× and 1.45× and DFlash with our objective at most 2.14× and 1.52× (Table G2).

W1. "does not measure DFlash's memory"

We measured it. At batch 1, DFlash adds 2.0 GiB to the original's 15.9 GiB, almost all of it the weights of its drafter, about 1.05B parameters. W = 4 has no drafter, adds 0.1 GiB, and its only new parameters are 0.5M (Table G6, Table 10 in Appendix E.2).

Q1. "How does IMLE-LLM compare with DFlash and EAGLE-3 at batch 4 and 16"

W = 4 is faster than DFlash at both sizes, 3.28× at batch 4 (DFlash 3.15×) and 2.86× at batch 16 (DFlash 2.67×). EAGLE-3 runs at 1.28× and 1.09× (Table G1, DFlash trained from scratch; the W = 4 cells are Table 6 of the submission, Table 9 in the revision).

W2. "The accuracy cost is larger than the retention numbers suggest"

W = 4 scores 95.65% of the original's math and 92.93% of its code, and the revised abstract now gives these next to 97.3% and 95.2% of the AR fine-tune, at the same 3.39× (L364-366 and L1337-1340 of the submission). The gap between the two pairs is the cost of fine-tuning, since the AR fine-tune itself loses 1.21, 1.83 and 1.66 points on GSM8K, HumanEval and IFEval. Our objective adds almost nothing to that cost, as our W = 4 weights decoded one token at a time are within 0.37 points of the AR fine-tune on all six tasks (Section 4.3).

W2 and Q3. "IFEval falls to 59.95 at W = 5" and "Why is IFEval affected much more than the other benchmarks"

The gap comes from a few strict-format rules. Without thinking, W = 4 loses 13.1 IFEval points on comma, letter-case and length rules and 2.5 on the other rule types, counted per instruction rather than per prompt, since one extra comma, capital letter or word fails a rule (Table 13 in Appendix G.2). With thinking on, the gap to the original narrows from 10.84 points to 3.9 (Table G4, Table 3 in Section 4.2). The three diffusion decoders of Table 1 built on Qwen3-8B lose 15.16 to 18.48 IFEval points against the original, while W = 4 loses 10.84. The 59.95 at W = 5 comes with 4.09×, and W = 4 scores 71.23 at 3.39×.

W2. "All results use one backbone in non-thinking mode"

We retrained W = 4 on Qwen3-8B's own thinking answers, with answers up to 32,768 tokens. It gets 94.3% of the original's six-task accuracy at 3.46×, while DFlash retrained on the same traces runs at 2.49× (Table G4, Section 4.2, Appendix B.3). W = 4 commits W tokens per window, so no draft needs accepting. On Llama-3.1-8B-Instruct it runs at 3.38× and scores 96.51% of the original's math and 92.02% of its code (Table G5, Appendix G.3).

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for the time and effort spent reviewing our paper. As the discussion period ends soon, we would be grateful to know whether our response has addressed the reviewer's concerns. If it has, we kindly ask the reviewer to consider raising the score. If any concern remains, we would be happy to address it in the time left.

I thank the authors for the detailed response and new experimental results, which addressed most of my questions. To reflect this, I have raised my rating to 3.

My remaining (minor) concern is that the core design, decoding without a drafter or verifier, is not clearly preferable in the main setting: as shown in Table 1, the proposed method (when W=4) is both lossy and slower on average than the lossless DFlash. Its lossless speed-up comes from using a separately trained drafter, which reintroduces the component the method was designed to avoid. The benefit of the core design is therefore limited to specific settings such as thinking mode and limited-memory scenarios.

AuthorsOpenReview ↗

Thank you, Reviewer dzDp

Thank you for reading our response so carefully and for raising the rating. We appreciate the reviewer’s time and the helpful feedback throughout. The point on scope is well taken, and we will make it clear in the abstract of the final version.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3), summary of the new results

We thank the reviewers. They found sharing one latent across a window "a clean way to couple tokens within a single pass", with "careful evaluation" (dzDp), the paper "very clearly written" on "a crucial problem" (KT3r), and adding a latent to an existing LLM without training from scratch "very impressive" (qmwp).

  • With our objective, DFlash's drafter runs at 4.75× at block 16 and 6.23× at block 32, against 3.61× and 3.62× with DFlash's own loss. All four drafters are trained from scratch on DFlash's recipe, and the output is exact because Qwen3-8B checks every draft (Table G1; dzDp, KT3r).
  • With thinking on and answers of up to 32,768 tokens, W = 4 scores 94.3% of the original's accuracy over six tasks, at 3.46× at batch 1. Its IFEval drop from the original is 3.9 points, vs. 10.84 without thinking (Table G4; qmwp, dzDp).
  • On Llama-3.1-8B-Instruct, a second model family, W = 4 runs at 3.38× with 96.51% math retention and 92.02% code retention (Table G5; qmwp, dzDp).
  • Eight baselines now run on Qwen3-8B, with EAGLE-3, I-DLM, Jacobi Forcing and K-Forcing added. Of the lossy decoders in Table G3, W = 4 is the fastest at 3.39× and has the highest code retention, 92.93% (Tables G1 and G3; qmwp, KT3r).
  • If the output must be exact, DFlash with our objective is the fastest decoder at batch 1, 4 and 16 (Table G1). If a lossy decoder is acceptable, W = 4 is the faster choice on open-ended text, at 3.54× on chat and 3.51× on IFEval, while the released DFlash gets 1.89× and 1.45× and DFlash with our objective at most 2.14× and 1.52× (Tables G1 and G2; dzDp).
  • With no drafter to serve, W = 4 adds 0.1 GiB to the original's serving memory at batch 1, where DFlash adds 2.0 GiB. The W = 4 fine-tune takes 699.27 H100-hours, close to the 632.46 of DFlash's recipe (Table G6; qmwp, dzDp, KT3r).
  • Speed and accuracy share one reference, the original Qwen3-8B. W = 4 keeps 95.65% of math and 92.93% of code at 3.39×, and 97.3% and 95.2% of the AR fine-tune (Table 1 and the revised abstract; dzDp).

Tables G1 to G6 follow in the next two comments.

AuthorsOpenReview ↗

General response (2/3), Tables G1, G2 and G3

Setting for all tables. Every method gets the same prompts and one grader per task, and released checkpoints use their authors' inference code. The new runs train on 8 H100, and each method is timed alone on one L40S, with speed-up over one-token decoding of the same base model. In Tables G1 to G3, decoding is greedy unless a column names a temperature, with thinking off, at batch 1 unless a column names another batch size, and answers on the six tasks of Table 1 stop at 4,096 new tokens.

Table G1. Our objective inside DFlash's drafter, and other drafters (revised PDF Table 2, Section 4.2)

DrafterAccepted, greedyAccepted, temp. 0.7Accepted, temp. 1.0Speed-up, batch 1Speed-up, batch 4Speed-up, batch 16
EAGLE-32.162.092.061.33×1.28×1.09×
I-DLM (exact mode)2.232.192.161.85×1.79×1.54×
DFlash, from scratch, block 164.494.023.803.61×3.15×2.67×
DFlash with our objective, from scratch, block 165.865.325.094.75×4.13×3.50×
DFlash, from scratch, block 324.614.133.893.62×3.08×2.41×
DFlash with our objective, from scratch, block 327.987.156.886.23×5.28×4.13×
Ours, W = 4 (no drafter, lossy)3.883.883.883.39×3.28×2.86×

Setting. The temperature 0.7 and 1.0 columns use the speculative-sampling rule with top-p 0.8. Accepted counts the tokens kept per check, including the one the check adds. For the W = 4 row it counts tokens per pass, 3.88 at every temperature. We trained all four DFlash rows from scratch on DFlash's recipe (800K samples, 6 epochs, LR 6e-4, same architecture), two with DFlash's loss and two with ours. Our rows differ from the DFlash row of the same block size only in the objective, with 8 latent draws in training, and in the latent projection Wz\mathbf{W}_z (+0.5M parameters). EAGLE-3 runs its public head, and I-DLM runs its exact mode at stride 4, which checks against I-DLM's own base.

Table G2. Speed-up by text type (revised PDF Table 12, Appendix G.1)

MethodChatIFEval
EAGLE-31.15×1.09×
DFlash, block 161.89×1.45×
DFlash with our objective, from scratch, block 162.07×1.52×
DFlash with our objective, from scratch, block 322.14×1.52×
Ours, W = 43.54×3.51×

Setting. DFlash runs its public block-16 checkpoint, as in Table 1, and the rows with our objective are the drafters of Table G1. Chat is the mean of MT-Bench, AlpacaEval 2 and Arena-Hard.

Table G3. Lossy decoders on Qwen3-8B (revised PDF Table 11 and Figure 8, Appendix F)

MethodMath retentionCode retentionIFEvalSpeed-up, batch 1
SDAR-8B91.34%80.31%63.591.77×
I-DLM-8B97.53%91.08%80.222.26×
ReFusion90.03%84.81%65.622.35×
Jacobi Forcing94.89%88.55%67.652.44×
WeDLM-8B93.29%83.35%66.912.72×
K-Forcing88.06%82.56%59.522.85×
Ours, W = 495.65%92.93%71.233.39×

Setting. Retention is the score relative to the original Qwen3-8B. I-DLM-8B runs at stride 4, its paper's default, where it commits 2.69 tokens per pass, and it checks drafts with its own fine-tuned weights. We trained Jacobi Forcing (with its authors' released code) and K-Forcing on Qwen3-8B with the corpus of Appendix B.1, each with at least the training compute of our W = 4 run. Rows that decode with noise (W = 4, K-Forcing) average three seeds, with a standard deviation of at most 0.4 points.

AuthorsOpenReview ↗

General response (3/3), Tables G4, G5 and G6, and the changes in the revised PDF

Table G4. Thinking mode on Qwen3-8B (revised PDF Table 3, Section 4.2 and Appendix B.3)

ModelMATH-500AIME 2024AIME 2025GPQA-DiamondLiveCodeBenchIFEvalSpeed-up, batch 1Speed-up, batch 4
Original Qwen3-8B97.075.066.561.651.584.7n/an/a
AR fine-tune96.774.265.861.251.084.1n/an/a
Ours, W = 494.869.060.059.648.180.83.46×3.38×
DFlash, retrained, block 16------2.49×2.17×
DFlash with our objective, block 16------3.29×2.89×
DFlash with our objective, block 32------4.33×3.66×

Setting. Training uses Qwen3-8B's own thinking answers to the training prompts of Appendix B.1, up to 32,768 tokens each. W = 4 and the AR fine-tune train on as many tokens as the submitted W = 4 run, and the drafters use DFlash's recipe. Decoding uses temperature 0.6, top-p 0.95 and top-k 20, the mean of 8 samples (16 for AIME) and a 32,768-token budget. The DFlash rows are exact under the speculative-sampling rule, and a dash stands for the original's accuracy.

Table G5. Llama-3.1-8B-Instruct (revised PDF Tables 14 and 15, Appendix G.3)

RowSpeed-upGSM8KMATH-500HumanEvalMBPPLiveCodeBenchIFEvalMath retentionCode retention
Originaln/a84.6150.8071.9570.6311.6775.05100%100%
AR fine-tunen/a82.7149.4070.7369.5811.4573.5797.50%98.30%
Ours, W = 43.38×81.9148.8765.8566.4010.5766.3696.51%92.02%

Setting. W = 4 and the AR fine-tune use the submitted recipe of Appendix B.2, with the same prompts and Llama's own answers as targets. The AR fine-tune uses the next-token loss. W = 4 averages three noise seeds and is timed at batch 1 against one-token decoding of the same model. Retention is the mean of per-task ratios to the original, with math over GSM8K and MATH-500 and code over HumanEval, MBPP and LiveCodeBench.

Table G6. Training cost and memory on Qwen3-8B (revised PDF Tables 5 and 10, Appendices B.2 and E.2)

Original Qwen3-8BOurs, W = 4DFlashDFlash with our objective
Training, H100-hoursn/a699.27632.46743.83
Training datan/aabout 100K prompts, 8 epochs800K samples, 6 epochs800K samples, 6 epochs
Serving peak, batch 115.9 GiB16.0 GiB17.9 GiB17.9 GiB
Serving peak, batch 1625.5 GiB26.5 GiB28.7 GiB28.7 GiB

Setting. Training hours are for one node of 8 H100. The DFlash column is DFlash's own block-16 recipe, and the column with our objective is the block-16 run of Table G1. Serving peaks are measured with 4K-token answers. When serving, DFlash with our objective uses one latent per block. The 8 draws happen only in training.

Changes in the revised PDF (marked in blue)

ChangeWhereAsked by
Retention against the original next to the AR fine-tune figuresAbstractdzDp
Five steps of one training round, with a pointer to Figure 5Section 3.2qmwp, KT3r
Table G1 and a "when to use which decoder" paragraphSection 4.2, Table 2, Appendix B.4dzDp, KT3r, qmwp
Thinking mode (Table G4)Section 4.2, Table 3, Appendix B.3qmwp, dzDp
Llama-3.1-8B-Instruct (Table G5)Section 4.2, Appendix G.3qmwp, dzDp
Training cost and serving memory (Table G6)Section 4.2, Appendices B.2 and E.2qmwp, dzDp, KT3r
Lossy decoders on Qwen3-8B (Table G3) and a speed-accuracy figureSection 4.2, Appendix Fqmwp, KT3r
Five Table 1 accuracy cells rerun by us (see the reply to qmwp), and the sentence on decoders "built on the same Qwen3-8B" limited to WeDLM, ReFusion and SDARTable 1, Sections 1 and 4.1qmwp
One-token scores of W = 4 on MBPP, LiveCodeBench and IFEvalSection 4.3dzDp
I-DLM cited, and our design compared with K-ForcingSection 5KT3r, qmwp
Speed by text type (Table G2) and IFEval by rule familyAppendices G.1 and G.2dzDp
Thinking mode and the tested backbones in the LimitationsSection 6dzDp, qmwp
DBLast and FlashMap cited as concurrent workAppendix HdzDp, KT3r

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.