- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ForkLeft, an AR-to-DLM distillation method that addresses the mismatch between causal AR teachers and bidirectional DLM students. It uses entropy-first rollouts to expose uncertain token decisions and prefix-aligned distillation to supervise the teacher and student under the same left context. Experiments show improvements across ten benchmarks while retaining the DLM's native confidence-first decoding procedure at inference.
Critical strengths
- Addresses the AR–DLM structural mismatch. ForkLeft explicitly aligns the teacher's and student's conditioning contexts, avoiding comparisons between token distributions based on different information.
- Effective combination supported by ablations. Figure 4 shows that combining entropy-first rollouts with prefix-aligned supervision outperforms either change alone. The method also improves EDLM-4B on all ten benchmarks.
Critical weaknesses
Potential loss of bidirectional generation capability.
Prefix-aligned distillation trains the student using only left context, potentially reducing its ability to exploit bidirectional context during native parallel decoding. This is not directly evaluated.
Limited novelty.
The two identified failure modes have already been discussed in prior work, including The Flexibility Trap (Ni et al., 2026) and OPDLM (Su et al., 2026). The novelty lies mainly in combining existing insights.
Limited applicability beyond math tasks.
The method relies on final-answer verification for correctness-based supervision routing, making its applicability to tasks without verifiable answers unclear.
Limited model and block-size diversity.
All evaluated students are block-based DLMs, with a block size of 32 reported for the main experiments. The method is not evaluated on full-sequence DLMs such as LLaDA or Dream, nor is block-size sensitivity analyzed.
Poor figure readability.
Figures 4 to 7 emphasize relative improvements rather than absolute accuracies, making the results less intuitive to interpret.
Questions for authors
- Could the authors address the concerns raised above?
- How does entropy-first rollout compare with random-order and low-confidence-first rollouts under identical training conditions? This would help clarify whether the gains specifically come from entropy-based ordering or simply from prioritizing uncertain tokens.
2 comments
Thank you for the additional experiments.
However, I remain concerned about the effectiveness of the main method. The new experiments show a severe performance collapse, while native confidence-first performs well. This discrepancy remains unexplained and raises doubts about the robustness of the proposed method.
Furthermore, the limited generalizability remains unaddressed.
Therefore, I will maintain my original rejection rating.
Response to Reviewer d65t: Right-Context Evaluation, Contribution, Rollout-Order Comparison
We thank the reviewer for the concrete questions; Q1 is answered point by point (W1 to W5), then Q2.
We trained twelve matched control arms: Efficient-DLM-4B, 600 updates, identical questions, supervised positions, optimizer and seed; teacher Qwen2.5-Math-7B-Instruct; eight of the ten benchmarks; one run per arm, so macro differences below 1 pp are not interpreted. The reference arm, ForkLeft (native-order configuration), keeps correctness routing, the prefix-aligned content loss and the ranking loss, with the sampler's native confidence-first rollout order. Results are compared within the series (Table 1: 2,000 updates, 30B teacher).
W1. Bidirectional capability: "This is not directly evaluated." We evaluated it directly: ForkLeft training does not remove the model's use of right context. Token accuracy on 6,000 positions (500 questions) under an identical left prefix, with the right context real, masked or shuffled:
Masking the right context costs ForkLeft 11.18 pp (base: 9.47 pp) and shuffling it 48.30 pp. This is expected: left-only prefixes enter only the loss at sampled positions; training rollouts and all inference use the native two-sided sampler. Parallel decoding on MATH500 (correct of 500):
At 32 and 16 passes ForkLeft is within 2 pp of the base model; at 8 and 4 passes both drop and ForkLeft drops further, so we claim equal inference cost and do not claim improved few-pass decoding.
W2. "The novelty lies mainly in combining existing insights." The two failure modes were identified by Ni et al. and Su et al. and are credited; what is new is separating rollout order from supervision view and showing that the view carries the larger effect. Teacher and student are read out under the same completed student prefix, which Reviewer c2b7 calls "a meaningful change from OPDLM". Figure 4's 2×2 (ten-benchmark macro, %; base 40.76):
The view is worth 4.79 pp (10 of 10 benchmarks) and 2.80 pp (9 of 10); the order 2.62 pp and 0.64 pp. Plain SFT shows the same view effect in the new controls: 48.97% under left prefixes vs 38.54% under native masking.
W3. "applicability to tasks without verifiable answers unclear." Training uses checkable answers, and the gains transfer outside math: in Table 1, MMLU rises from 71.04% to 76.05% (+703 items), MLogiQA by 7.50 pp, CEval by 2.97 pp and GPQA-D by 1.51 pp. The new routing ablation narrows the verifier's role: the arm that keeps the correctness gate but never trains on reference answers is the best of the series (49.84%). The method needs only a correct/incorrect verdict, which covers exact-match, multiple-choice and executable-check tasks. Tasks with no checkable outcome are outside the present scope; the limitations will say so.
W4. "not evaluated on full-sequence DLMs such as LLaDA or Dream, nor is block-size sensitivity analyzed." The paper covers two student families and two sizes with their native decoders: EDLM-4B (+4.93 pp, 10 of 10 benchmarks), EDLM-8B (+3.22 pp, 9 of 10) and SDAR-4B (+1.32 pp after only 500 updates, 7 of 10 with one tie). All use 32-token blocks; the W1 table varies the passes within a block. Full-sequence students and other block sizes are outside the current evidence; the limitations will say so.
W5. "Figures 4 to 7 emphasize relative improvements rather than absolute accuracies." We agree and will redraw Figures 4 to 7 with absolute accuracy as the primary quantity and gains as annotations.
Q2. "How does entropy-first rollout compare with random-order and low-confidence-first rollouts under identical training conditions?" We ran all three, plus the native order and strict left-to-right; only the training rollout's commit order differs (accuracy %):
Entropy-first is within 1 pp macro of lowest-confidence-first and random, and above random on GSM8K (80.29% vs 73.09%). At this budget only the native order keeps iterative generation, so prioritizing uncertain tokens alone is not sufficient. The Table 1 run uses entropy-first rollouts, which at the full budget are 2.62 pp above the low-entropy order (W2). We will report both budgets side by side.