acceptodds
Under review as a conference paper at ICLR 2027

ForkLeft: Entropy-First Rollouts for Prefix-Aligned Autoregressive-to-Diffusion Distillation

Abstract

Autoregressive Next-Token Prediction (NTP) has enabled strong reasoning capabilities in language models, while Diffusion Language Models (DLMs) offer flexible token orders and parallel generation. We ask whether DLMs can acquire NTP-style reasoning through distillation without giving up their native generation process. Direct distillation, however, faces a fundamental mismatch: an autoregressive teacher predicts from a left prefix, whereas a DLM can condition on tokens on both sides. We introduce ForkLeft, a distillation framework that resolves this mismatch by separating the student's rollout from teacher supervision. During training, the student first performs entropy-first rollouts that commit uncertain positions and expose potential forks. We then fix the resulting student prefix and distill an NTP teacher under the same context, with answer correctness determining the supervision source. At inference, the student returns to its native confidence-first parallel decoding. With Qwen3-30B-A3B-Base, ForkLeft improves Efficient-DLM-4B on all ten benchmarks, raising MATH500 from 72.60% to 79.60%, and consistently outperforms three alternative designs. The gains scale with teacher strength and generalize to SDAR-4B with only 500 updates. At matched scale, the distilled 4B and 8B students exceed the published SDAR-Chat and OPDLM models on seven benchmarks, showing that DLMs can learn NTP-style reasoning without sacrificing native parallel generation. Code and datasets will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Knowledge Distillation, On-Policy Distillation, Autoregressive Models, Masked Diffusion, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d65tOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ForkLeft, an AR-to-DLM distillation method that addresses the mismatch between causal AR teachers and bidirectional DLM students. It uses entropy-first rollouts to expose uncertain token decisions and prefix-aligned distillation to supervise the teacher and student under the same left context. Experiments show improvements across ten benchmarks while retaining the DLM's native confidence-first decoding procedure at inference.

Critical strengths
  • Addresses the AR–DLM structural mismatch. ForkLeft explicitly aligns the teacher's and student's conditioning contexts, avoiding comparisons between token distributions based on different information.
  • Effective combination supported by ablations. Figure 4 shows that combining entropy-first rollouts with prefix-aligned supervision outperforms either change alone. The method also improves EDLM-4B on all ten benchmarks.
Critical weaknesses

Potential loss of bidirectional generation capability.

Prefix-aligned distillation trains the student using only left context, potentially reducing its ability to exploit bidirectional context during native parallel decoding. This is not directly evaluated.

Limited novelty.

The two identified failure modes have already been discussed in prior work, including The Flexibility Trap (Ni et al., 2026) and OPDLM (Su et al., 2026). The novelty lies mainly in combining existing insights.

Limited applicability beyond math tasks.

The method relies on final-answer verification for correctness-based supervision routing, making its applicability to tasks without verifiable answers unclear.

Limited model and block-size diversity.

All evaluated students are block-based DLMs, with a block size of 32 reported for the main experiments. The method is not evaluated on full-sequence DLMs such as LLaDA or Dream, nor is block-size sensitivity analyzed.

Poor figure readability.

Figures 4 to 7 emphasize relative improvements rather than absolute accuracies, making the results less intuitive to interpret.

Questions for authors
  • Could the authors address the concerns raised above?
  • How does entropy-first rollout compare with random-order and low-confidence-first rollouts under identical training conditions? This would help clarify whether the gains specifically come from entropy-based ordering or simply from prioritizing uncertain tokens.
2 comments
AuthorsOpenReview ↗

Response to Reviewer d65t: Right-Context Evaluation, Contribution, Rollout-Order Comparison

We thank the reviewer for the concrete questions; Q1 is answered point by point (W1 to W5), then Q2.

We trained twelve matched control arms: Efficient-DLM-4B, 600 updates, identical questions, supervised positions, optimizer and seed; teacher Qwen2.5-Math-7B-Instruct; eight of the ten benchmarks; one run per arm, so macro differences below 1 pp are not interpreted. The reference arm, ForkLeft (native-order configuration), keeps correctness routing, the prefix-aligned content loss and the ranking loss, with the sampler's native confidence-first rollout order. Results are compared within the series (Table 1: 2,000 updates, 30B teacher).

W1. Bidirectional capability: "This is not directly evaluated." We evaluated it directly: ForkLeft training does not remove the model's use of right context. Token accuracy on 6,000 positions (500 questions) under an identical left prefix, with the right context real, masked or shuffled:

CheckpointRealMaskedShuffled
Base0.97930.88470.3915
ForkLeft0.98120.86930.4982

Masking the right context costs ForkLeft 11.18 pp (base: 9.47 pp) and shuffling it 48.30 pp. This is expected: left-only prefixes enter only the loss at sampled positions; training rollouts and all inference use the native two-sided sampler. Parallel decoding on MATH500 (correct of 500):

Passes per 32-token block321684
Base36231318527
ForkLeft353310936

At 32 and 16 passes ForkLeft is within 2 pp of the base model; at 8 and 4 passes both drop and ForkLeft drops further, so we claim equal inference cost and do not claim improved few-pass decoding.

W2. "The novelty lies mainly in combining existing insights." The two failure modes were identified by Ni et al. and Su et al. and are credited; what is new is separating rollout order from supervision view and showing that the view carries the larger effect. Teacher and student are read out under the same completed student prefix, which Reviewer c2b7 calls "a meaningful change from OPDLM". Figure 4's 2×2 (ten-benchmark macro, %; base 40.76):

Rollout orderPrefix-aligned viewBidirectional view
Entropy-first45.6940.90
Low-entropy-first43.0740.27

The view is worth 4.79 pp (10 of 10 benchmarks) and 2.80 pp (9 of 10); the order 2.62 pp and 0.64 pp. Plain SFT shows the same view effect in the new controls: 48.97% under left prefixes vs 38.54% under native masking.

W3. "applicability to tasks without verifiable answers unclear." Training uses checkable answers, and the gains transfer outside math: in Table 1, MMLU rises from 71.04% to 76.05% (+703 items), MLogiQA by 7.50 pp, CEval by 2.97 pp and GPQA-D by 1.51 pp. The new routing ablation narrows the verifier's role: the arm that keeps the correctness gate but never trains on reference answers is the best of the series (49.84%). The method needs only a correct/incorrect verdict, which covers exact-match, multiple-choice and executable-check tasks. Tasks with no checkable outcome are outside the present scope; the limitations will say so.

W4. "not evaluated on full-sequence DLMs such as LLaDA or Dream, nor is block-size sensitivity analyzed." The paper covers two student families and two sizes with their native decoders: EDLM-4B (+4.93 pp, 10 of 10 benchmarks), EDLM-8B (+3.22 pp, 9 of 10) and SDAR-4B (+1.32 pp after only 500 updates, 7 of 10 with one tie). All use 32-token blocks; the W1 table varies the passes within a block. Full-sequence students and other block sizes are outside the current evidence; the limitations will say so.

W5. "Figures 4 to 7 emphasize relative improvements rather than absolute accuracies." We agree and will redraw Figures 4 to 7 with absolute accuracy as the primary quantity and gains as annotations.

Q2. "How does entropy-first rollout compare with random-order and low-confidence-first rollouts under identical training conditions?" We ran all three, plus the native order and strict left-to-right; only the training rollout's commit order differs (accuracy %):

Rollout orderMacro (8)MATH500GSM8K
Native confidence-first (reference)49.3170.6082.41
Entropy-first37.130.0080.29
Lowest-confidence-first37.050.2079.30
Strict left-to-right36.951.4078.24
Random36.381.6073.09

Entropy-first is within 1 pp macro of lowest-confidence-first and random, and above random on GSM8K (80.29% vs 73.09%). At this budget only the native order keeps iterative generation, so prioritizing uncertain tokens alone is not sufficient. The Table 1 run uses entropy-first rollouts, which at the full budget are 2.62 pp above the low-entropy order (W2). We will report both budgets side by side.

Thank you for the additional experiments.

However, I remain concerned about the effectiveness of the main method. The new experiments show a severe performance collapse, while native confidence-first performs well. This discrepancy remains unexplained and raises doubts about the robustness of the proposed method.

Furthermore, the limited generalizability remains unaddressed.

Therefore, I will maintain my original rejection rating.

Reviewer_c2b7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ForkLeft, a framework for transferring knowledge from an autoregressive teacher to a diffusion language model while retaining the student’s native decoding procedure. Its central idea separates how training responses are generated from how they are supervised: the student commits its highest-entropy predictions first, then teacher and student distributions are compared using the same completed left prefix. A correctness verifier selects among teacher distillation, reinforcement of the student’s own response, reference supervision, and exclusion. An additional ranking loss aims to shape confidence for subsequent decoding. At inference, the student returns to confidence-first diffusion generation. The main experiment improves Efficient-DLM-4B on ten benchmarks, including MATH500 from 72.60% to 79.60%; further experiments examine different teachers, an 8B student, and transfer to SDAR-4B. My main concern is that the paper shows promising gains for the full recipe, but doesn’t clearly show that they’re reproducible or that they come from the proposed distillation mechanism. First, key details are missing: the training data, its size and splits, optimizer settings, hyperparameters like the sampling fraction, ranking weight, and margins, and total compute. Second, the evidence that the teacher adds value beyond SFT is mixed. In Table 4, SFT alone beats OPD alone on every benchmark, and adding OPD helps six benchmarks but hurts four, leaving the average about 0.57 points below SFT. That doesn’t prove the combination is worse, but the claim that OPD “adds to SFT” should be toned down, especially since the comparison isn’t matched for prefixes, supervised tokens, or compute. Third, comparing high- and low-entropy rollouts isn’t the same as testing against the native confidence-first rollout, and routing and confidence ranking aren’t ablated separately. Finally, high entropy doesn’t necessarily mark an important reasoning decision, and the completed prefixes may contain tokens that weren’t available at the original commitment, so accuracy gains alone don’t show the method actually fixes these “forks.”

Critical strengths

The strongest contribution is the clear separation between the student’s rollout policy and the information available during distillation. Comparing teacher and student under a shared prefix is a reasonable response to their different conditioning structures. This represents a meaningful change from OPDLM, which supervises a student’s partially masked states using teacher distributions conditioned on completed causal prefixes. The four-way rollout-order × supervision-view experiment is particularly useful: it supports an interaction between entropy-first generation and prefix-aligned teaching, rather than merely showing that a collection of modifications improves a baseline. Several gains are substantial: MATH500 improves by 35 correct answers, MLogiQA by 60, and MMLU by 703. These deserve weight beyond the smaller benchmark improvements. The teacher comparisons and SDAR experiment provide additional evidence that the recipe has some breadth. I also appreciate the exact-count tables and the explicit acknowledgment of benchmark dependence, unmatched published-system comparisons, and limitations of the theoretical rationale.

Critical weaknesses

My main concern is that the paper demonstrates promising improvements for the complete recipe more convincingly than it establishes a reproducible advantage attributable to its proposed distillation mechanism. First, essential experimental details are missing. Section 4.1 and Appendix C do not identify the training dataset, its size and splits, reference-answer sources, or checks for benchmark overlap. They also omit the optimizer, learning rate, batch size, sampling fraction ρρ, ranking weight λrankλ_{\mathrm{rank}}, margins μμ and δδ, and total training tokens or compute. Consequently, “2,000 updates on math-domain questions” is insufficient to reproduce the results or assess the training budget. Second, the evidence that teacher distributions add value beyond SFT is mixed. In Table 4, SFT alone outperforms OPD alone on every benchmark. Combining SFT and OPD improves six benchmarks over SFT but reduces four; calculated from the reported counts using the paper’s equal-benchmark averaging, the combination is approximately 0.57 percentage points below SFT. This average is sensitive to the tiny AIME and LMB-H sets and does not establish that combining objectives is worse, but it requires a more qualified interpretation of the claim that OPD “adds to SFT.” The comparison also uses different prefixes and is not matched for supervised tokens or compute. Third, the ablation does not directly test the native rollout baseline: low entropy and maximum token probability are different ranking rules, as the paper itself acknowledges. Thus, high- versus low-entropy results do not establish superiority over the actual native confidence-first training rollout. Routing and confidence ranking also lack independent ablations. Fourth, the proposed mechanism remains insufficiently validated. High token entropy need not identify a consequential reasoning decision, and supervision uses completed prefixes that can contain tokens unavailable when the original commitment occurred. Accuracy improvements alone therefore do not show that the method repairs the exposed “forks.”

Questions for authors

The relationship between the question qq and tokenized prompt PP should be introduced together in the main text, rather than clarified later in Appendix A.1. Exact checkpoint identifiers and Base/Instruct suffixes should be used consistently, especially in the teacher tables. The reference to “Appendix C, Section F.3” in the limitations does not direct readers to the shared-tokenizer definition, which appears in Appendix A.1. Equation (8) excludes the final response position when no EOS is present; although this is explicitly acknowledged, the reason for that choice should be explained. Figure 2’s statement that routes share loss positions should also distinguish student positions ScS_c from reference positions SgS_g. Appendix F.1 needs a clearer description of the static probe’s data, checkpoint, and right-context construction. Finally, “keeps the inference cost and parallelism” should be phrased more narrowly as retaining the architecture and sampler unless supported by timing and denoising-step measurements.

1 comment
AuthorsOpenReview ↗

Response to Reviewer c2b7: Configuration Details, Matched SFT Comparison, Order, Ranking and Routing Ablations

We thank the reviewer for the precise reading and ran the missing controls.

W1. "essential experimental details are missing." We agree; Section 4.1 and Appendix C will gain a configuration table for every reported row.

  • Main run (Table 1): Efficient-DLM-4B student, Qwen3-30B-A3B-Base teacher, 2,000 updates, entropy-first rollout, ρ = 0.25, λ_rank = 0.1, μ = 0.15, δ = 0.05.
  • New control series (twelve matched arms, 600 updates): same student; teacher Qwen2.5-Math-7B-Instruct (greedy, verified solutions); 1,200 drawn questions (826 verified); AdamW, 3e-6; batch 8 ranks × 2 rows; seed 42; at most 12 supervised positions per row; λ_rank = 0.1; eight of ten benchmarks; one run per arm. Reference arm: ForkLeft (native-order configuration) = routing, prefix-aligned content loss and ranking loss, native confidence-first order. Results are compared within the series.

For every row the table will also give the dataset revision, size and split, reference-answer source, benchmark-overlap check, optimizer, learning rate, batch size, seed, supervised tokens and GPU-hours.

W2. "the evidence that teacher distributions add value beyond SFT is mixed." We adopt the reviewer's calculation and ran the comparison with prefixes, positions and budget matched. In Table 4, adding OPD to SFT raises six benchmarks (+179 items) and lowers four (−6 items), macro −0.57 pp. In the new matched arms only the teacher target changes: the full teacher distribution gives the best MATH500 of the series (74.00%, vs 72.20% for the teacher's argmax; macro 48.99% and 49.42%). Matched teacher-data left-to-right SFT reaches 48.97%; ForkLeft (49.31%) is higher on five of eight benchmarks, one tie. The text will replace "adds to SFT" by the per-benchmark statement.

W3. Native-rollout baseline; "Routing and confidence ranking also lack independent ablations." We ran the native order, four other orders, and independent ablations of ranking and routing (accuracy %; routing rows share the argmax target):

Training changeMacro (8)MATH500
Native confidence-first (reference)49.3170.60
Entropy-first37.130.00
Lowest-confidence-first37.050.20
Strict left-to-right36.951.40
Random36.381.60
No ranking loss37.7410.60
Teacher argmax, full routing49.4272.20
Teacher argmax, gate, no reference replay49.8473.00
Teacher argmax, no gate48.5973.20

At this budget the native order keeps iterative generation and the four other orders do not; the ranking loss is needed (−11.57 pp macro without it); and the two gated arms are the best of the series. No gate differs from full routing by two AIME2025 items (0.83 pp), so we describe the gate as retained and reference replay as optional. The Table 1 run uses entropy-first rollouts, which at the full budget are 2.62 pp above low-entropy-first (10 of 10 benchmarks). We will report both budgets, with the native order as an explicit row.

W4. "the proposed mechanism remains insufficiently validated." We agree and will state the mechanism at the level the evidence supports: prefix-aligned teaching at positions of the student's own rollout. This choice is supported directly: under the same rollout order, prefix-aligned teaching is 4.79 pp above bidirectional-view teaching (10 of 10 benchmarks), and the same teacher solutions train a usable student only under left prefixes (48.97%, vs 38.54% under native masking). The revision will describe high entropy as an uncertain prediction, not a verified reasoning decision; state that the completed prefix can contain tokens committed after the supervised position; and remove wording that accuracy gains show repaired forks.

Minor points. All seven will be fixed.

  1. q and P: P will be defined from q at its first use in Section 3.
  2. Checkpoints: full identifiers with Base/Instruct suffixes.
  3. Cross-reference: "Appendix C, Section F.3" becomes Appendix A.1.
  4. Equation (8): positions are indexed from zero, so the candidate set {1, …, L−1} contains the final realized token. A truncated response receives no EOS target because it did not terminate. We will state the indexing convention and this reason at the equation.
  5. Figure 2: the caption will distinguish S_c (losses on student responses) from S_g (reference supervision).
  6. Appendix F.1: we will state the probe's data, checkpoint and construction (prefix-only readout: tokens left of the target; target-masked bidirectional readout: also those to its right), and add a fully specified diagnostic (6,000 positions): masking the right context costs ForkLeft 11.18 pp of token accuracy (base 9.47 pp).
  7. "keeps the inference cost and parallelism": we adopt the narrower wording and add measurements: MATH500 at 32/16/8/4 passes per 32-token block, ForkLeft 353/310/93/6 of 500, base 362/313/185/27, same wall time.
Reviewer_6scnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a method distilling reasoning capabilities from autoregressive teacher models into diffusion language models. They note a potential challenge with distillation is the mismatch between autoregressive reasoning traces that are generated left-to-right, and the any-order MDLM training objective. The authors propose an approach called ForkLeft to address this mismatch. With ForkLeft, samples are generated with entropy-based decoding, the autoregressive teacher supervises selected positions under the student-generated left prefix. ForkLeft improves performance of base diffusion checkpoints.

Critical strengths

The paper highlights the challenges of distilling from autoregressive to diffusion models, and proposes a novel approach.

Critical weaknesses

Missing baselines: The clear missing baseline is left-to-right student rollouts. A natural way to distill autoregressive models into diffusion models is to sample and train on left-to-right rollouts. This has already been explored to some extent in [1]. Another baseline is to perform SFT with teacher generated data, and train the student on the left-to-right ordering. Additionally, the authors do not provide a matched comparison with [2].

Limited evidence that the method outperforms SFT: It appears that no matched comparison between ForkLeft and SFT is provided in the paper. Also, in Table 4, OPD performs worse than SFT on all ten benchmarks.

Missing experiments: No experiments are conducted that study the effect of ForkLeft on the model’s parallelism and any-order capabilities. It is possible that the confidence based decoding on this model reverts to a left-to-right autoregressive sampler.

Presentation: The paper repeatedly overstates results, and uses vague language across the results section and captions (e.g. “Teacher distributions add to hard targets on the same questions”). Also, figures and labels can be challenging to read and parse.

[1] Ni et al. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. ICML, 2026.

[2] Su et al. Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation. arXiv, 2026.

Questions for authors

Do you have any evidence that you did not include that compares your approach with SFT?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6scn: New Left-to-Right and Matched-SFT Baselines, Parallel-Decoding Measurements

We thank the reviewer for naming specific baselines; we ran each one.

We trained twelve matched control arms: Efficient-DLM-4B, 600 updates, identical questions, supervised positions, optimizer and seed; teacher Qwen2.5-Math-7B-Instruct; eight of the ten benchmarks; one run per arm, so macro differences below 1 pp are not interpreted. The reference arm, ForkLeft (native-order configuration), keeps correctness routing, the prefix-aligned content loss and the ranking loss, with the sampler's native confidence-first rollout order. Results are compared within the series (Table 1: 2,000 updates, 30B teacher).

ArmMacro (8)MATH500GSM8K
Base48.8672.4082.79
ForkLeft49.3170.6082.41
Left-to-right student rollouts36.951.4078.24
Teacher-data SFT, left-to-right48.9772.8082.26
Teacher-data SFT, native masking38.5413.2075.28

W1a. "The clear missing baseline is left-to-right student rollouts." We ran it: ForkLeft is higher on 8 of 8 benchmarks (macro +12.36 pp). Only the training rollout's commit order changes. Iterative generation is lost (MATH500 1.40%), while the single-pass multiple-choice benchmarks move by at most 1.01 pp. We will add the arm and discuss it with Ni et al. [1].

W1b. "SFT with teacher generated data, and train the student on the left-to-right ordering." We ran it under matched conditions: ForkLeft is higher on five of eight benchmarks with one tie (macro 49.31% vs 48.97%). The SFT arm is cross-entropy on verified teacher solutions under left-only prefixes, same rows, positions and updates. It is higher on MATH500 (+2.20 pp) and MLogiQA (+0.38 pp), so we describe ForkLeft as on par with matched SFT at this budget and ahead on more benchmarks. The same teacher solutions under native masking collapse (38.54%): the left-prefix view is what makes teacher data usable, as in Figure 4.

W1c. "the authors do not provide a matched comparison with [2]." Under one prompt template and scorer, ForkLeft is above the published OPDLM models on 8 of 10 benchmarks at 4B and 7 of 10 at 8B (one level, one lower at 4B; one level, two lower at 8B), e.g. MATH500 +6.80 / +10.00 pp and MMLU +10.55 / +8.21 pp. Within our pipeline, changing only the supervision view (Figure 4), prefix-aligned teaching is 4.79 pp above bidirectional-view teaching (10 of 10 benchmarks). We did not retrain the full OPDLM recipe and will label the table as a same-scale comparison of published systems.

W2 and the question. "Do you have any evidence that you did not include that compares your approach with SFT?" Yes: the matched ForkLeft-vs-SFT run of W1b, new in this response, plus the two points below.

  1. Table 4: the reviewer's reading ("OPD performs worse than SFT on all ten benchmarks") is correct and will be stated. Table 4 isolates the content loss on teacher-correct/student-wrong questions; it is not the full method. Both objectives improve on the base model (SFT +4.48 pp, OPD +3.16 pp); adding OPD to SFT raises six benchmarks (+179 items) and lowers four (−6 items), macro −0.57 pp.
  2. Teacher targets at matched positions (new): the full teacher distribution gives the best MATH500 of the series (74.00%, vs 72.20% for the teacher's argmax); macro 48.99% and 49.42%.

W3. "reverts to a left-to-right autoregressive sampler." We measured this: the trained model still relies on right context and decodes with the native sampler at the same cost. Token accuracy on 6,000 positions (500 questions) under an identical left prefix, with the right context real, masked or shuffled:

CheckpointRealMaskedShuffled
Base0.97930.88470.3915
ForkLeft0.98120.86930.4982

Masking the right context costs ForkLeft 11.18 pp (base: 9.47 pp) and shuffling it 48.30 pp; a left-to-right predictor would be insensitive to both. Parallel decoding on MATH500 (correct of 500):

Passes per 32-token block321684
Base36231318527
ForkLeft353310936

At 32 and 16 passes ForkLeft is within 2 pp of the base model at the same wall time. At 8 and 4 passes both drop and ForkLeft drops further, so we claim equal inference cost and do not claim improved few-pass decoding.

W4. Presentation ("overstates results", "vague language", figures "challenging to read"). We will replace the caption the reviewer quotes by the measured statement of W2, narrow the inference claim to "same architecture and native sampler, no teacher or verifier at inference", and redraw the figures with absolute accuracy, counts and larger labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.