acceptodds
Under review as a conference paper at ICLR 2027

Hesitation-Aware On-Policy Distillation for Diffusion Language Models

Abstract

Diffusion large language models (dLLMs) generate text by iterative unmasking. At each denoising step, a dLLM proposes a token at every masked position, but the decoder commits only a confident subset of these proposals. Trace-based on-policy distillation (TOPD) builds on this process by matching the student to a stronger teacher, yet only at the committed positions. We argue that this discards much of the useful signal, which resides in the uncommitted proposals, where the student has made a prediction but is not yet confident enough to commit it. We call these proposals hesitations. In our pilot study on an SDAR-4B student, hesitations make up only 24% of supervisable state–position pairs but carry 66% of the teacher–student divergence. To exploit this signal, we propose (HOPD), which extends teacher distribution matching to every masked position of each denoising step. Because hesitations are not equally informative, we further allocate supervision using hindsight from the completed trajectory, placing more weight on positions whose proposals often disagree with the final token and on blocks whose first-step proposals rarely survive. Since both models already produce distributions at all masked positions, HOPD requires no additional forward passes over TOPD. The only extra cost is evaluating the loss at more positions. With SDAR-1.7B and SDAR-4B students distilled from TraDo-8B-Instruct, HOPD achieves the best average score among the evaluated methods on five math and coding benchmarks, under both static and dynamic decoding and at both scales. It also speeds up decoding: on SDAR-4B, the HOPD student hesitates less and commits 11% more tokens per denoising step than TOPD, while reaching higher accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, On policy distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E7ngOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HOPD, an on-policy distillation method for diffusion language models. It applies teacher supervision to all masked positions, including predictions the student has not yet committed. It also uses the completed decoding trajectory to give more weight to unstable predictions. Experiments show improved average accuracy on math and coding tasks, along with faster decoding on MATH500.

Critical strengths
  1. The idea is simple and well motivated. HOPD uses teacher guidance for token predictions that TOPD leaves out, without extra model forward passes.
  2. The experiments cover math and coding, different model sizes, and both static and dynamic decoding. The results show improvements in accuracy and decoding speed.
Critical weaknesses
  1. The paper does not show why hindsight-based hesitation is a better weighting signal than token confidence or entropy. Token changes may also reflect sampling noise or valid alternatives.
  2. The paper uses evaluation benchmarks to select checkpoints (Appendix B.4, lines 848–851). This may lead to overly optimistic results on those benchmarks.
Questions for authors
  1. Do the hindsight weights outperform weights based on token confidence or entropy?
  2. The paper argues that deferred and retracted predictions need different treatment. How does HOPD address both needs, and what evidence shows the benefit of supervising each group?
1 comment
AuthorsOpenReview ↗

Response to Reviewer E7ng

We thank the reviewer for the helpful comments. We address the concerns as follows.

W1 & Q1. Hindsight weights versus confidence-based weights

Thank you for this question. We trained a variant of HOPD that replaces the hindsight weights with confidence-based weights 1 − c, where c is the student's confidence in its proposed token, and keeps everything else unchanged. The results is given by in the following.

ModelWeightsMATH500AIME24GSM8KLCB-v2LiveBenchAvg.
SDAR-4BConfidence (1 − c)75.1/72.58.5/10.591.2/90.222.0/19.922.1/18.043.8/42.2
Hindsight (HOPD)74.9/72.515.0/9.791.1/90.720.9/19.822.1/19.844.8/42.5
SDAR-1.7BConfidence (1 − c)63.2/58.43.5/4.282.4/78.311.2/8.28.3/8.333.7/31.5
Hindsight (HOPD)66.8/59.53.7/5.083.6/78.811.1/8.910.9/9.935.2/32.4

HOPD outperforms the confidence-based variant on the five-benchmark average in all four settings, by 1.0/0.3 on SDAR-4B and 1.5/0.9 on SDAR-1.7B. The hindsight weights are therefore the more effective signal. The reason is that confidence, like entropy, is measured at a single step and cannot tell whether a proposal will later be kept or overturned. A deferred prediction, a low-confidence proposal that is later committed unchanged, receives a large weight, while a retracted prediction may receive a small one.

W2. Checkpoint selection

Thank you for raising this point. The selection rule in Appendix B.4 is the same for all methods. To remove its effect, we evaluated every TOPD and HOPD model at round 30, the last round of our fixed training budget, without using any benchmark to choose the checkpoint. Entries are static/dynamic.

ModelMethodMATH500AIME24GSM8KLCB-v2LiveBenchAvg.
SDAR-4BTOPD73.7/71.313.3/9.091.2/89.921.9/17.719.8/17.244.0/41.0
HOPD73.9/71.412.3/7.891.4/90.420.4/18.322.1/18.844.0/41.3
SDAR-1.7BTOPD62.3/57.14.2/2.381.7/79.19.3/7.29.4/6.533.4/30.4
HOPD63.0/58.11.0/6.083.6/79.811.1/8.910.9/9.933.9/32.6
LLaDA-8BTOPD37.0/37.30.2/0.083.5/83.27.5/6.76.8/6.227.0/26.7
HOPD39.0/38.10.2/0.282.2/82.48.0/6.310.9/7.028.1/26.8

Without selection, HOPD is never below TOPD in the six model–decoding settings. The gain is largest on SDAR-1.7B, +0.5/+2.2, and on LLaDA-8B under static decoding, +1.1. On SDAR-4B, the two methods reach comparable accuracy, but HOPD decodes faster. The table below gives the tokens committed per denoising step under dynamic decoding.

ModelMethodMATH500AIME24GSM8K
SDAR-4BTOPD2.231.891.97
HOPD2.462.092.22
SDAR-1.7BTOPD1.951.601.76
HOPD2.352.022.24

HOPD commits 10–12% more tokens per step on SDAR-4B and 20–28% more on SDAR-1.7B. Therefore, HOPD's advantage does not depend on checkpoint selection. It is more accurate on SDAR-1.7B and LLaDA-8B, and on SDAR-4B it reaches comparable accuracy while decoding faster.

Q2. Handling deferred and retracted predictions

Thank you for this question. HOPD supervises both groups with the same loss, but treats them differently in the hindsight weights. The loss matches the teacher's distribution at each position, so its effect depends on the teacher. Where the teacher is confident, the student can learn to commit a deferred proposal earlier, and where the teacher is uncertain, a retracted position stays uncertain. In the weights, a deferral matches the final token and adds no weight, while a retraction raises both the position and block weights.

To test each group, we removed it from the uniform target set. Entries are the math three-benchmark average, static/dynamic.

ModelUniform (all)w/o deferredw/o retractedCommitted onlyHOPD
SDAR-4B59.3/57.557.5/56.356.8/55.658.1/57.060.3/57.6
SDAR-1.7B51.0/46.749.3/46.648.4/47.050.0/46.051.4/47.8

Overall, supervising both groups improves the average over removing either of them, and HOPD gives the best results in all four settings. Therefore, both groups contribute, and weighting them by hindsight further improves the results.

Reviewer_dWMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hesitation-Aware On-Policy Distillation (HOPD) to improve the training of diffusion language models. Unlike previous methods that only supervise committed tokens, HOPD also uses predictions that are not committed during the denoising process. The authors show that these uncommitted predictions account for a large portion of the disagreement between the student and teacher models. Based on this observation, HOPD applies teacher supervision to all masked positions and introduces a hindsight weighting strategy to emphasize uncertain predictions.

Critical strengths
  1. The motivation is clear and well supported by experimental evidence.

  2. The method is very simple and easy to implement.

  3. The proposed approach improves both accuracy and decoding speed at the same time.

  4. The paper is well written and easy to follow.

Critical weaknesses
  1. Hesitation supervision is the core novelty of the paper, but the experiments do not adequately isolate its own contribution. Compared with TOPD, HOPD not only adds supervision on uncommitted positions but also changes the KL objective and the training weights. The authors need to provide comprehensive results showing that hesitation supervision alone is effective.

  2. The hindsight weighting is heuristic, and its gains are very limited. It is unclear whether this heuristic actually works across different dLLM models.

  3. The main experiments only have experimental results on SDAR-1.7B/4B, and on LLaDA HOPD improves average accuracy by only about 0.2%, which is too marginal. The paper should include more experiments showing that HOPD achieves consistent and significant accuracy gains on more advanced dLLMs, including LLaDA 2.1, DMax, and Fast-dllm-v2.

  4. The experiments on SDAR-4B-Chat in Appendix B.6 show that when both HOPD and TOPD use full-vocab distillation, HOPD's accuracy advantage drops to only 0.3%. This makes me seriously question whether the proposed hesitation supervision and hindsight weighting are actually effective.

Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dWMj

We thank the reviewer for the valuable comments. We address the concerns as follows.

W1 & W4. Isolating the effect of hesitation supervision

Thank you for this important question. Our ablations change one component at a time, so the effect of hesitation supervision can be separated from the objective and the weights. Entries are the five-benchmark average, static/dynamic, from Tables 1, 5, 10, 13 and 15. The four settings are the two models under static and dynamic decoding.

ModelTOPD (sampled-token)+ full-vocab KL+ hesitation targets (uniform)+ hindsight weights (HOPD)
SDAR-4B43.9/40.943.8/42.244.1/42.044.8/42.5
SDAR-1.7B33.7/30.733.6/30.934.5/31.635.2/32.4
  • Hesitation supervision alone. With the objective fixed and uniform weights, adding the hesitation targets improves the average in three of the four settings, by 0.4 points on average. On math, it improves all four settings.
  • The hindsight weights improve the average further in all four settings, by 0.7 points on average.
  • The objective contributes 0.3 points on average, mostly on SDAR-4B with dynamic decoding.

Regarding W4, the 0.3-point gap is the smallest of the four settings. Even there, the hindsight weights raise the average from 42.0 to 42.5, and in the other three settings the two components add 1.0 to 1.6 points over full-vocabulary TOPD. Therefore, both hesitation supervision and the hindsight weights are effective, and together they account for about three quarters of HOPD's gain over TOPD.

W2. Hindsight weights

Thank you for this comment. Although the hindsight weights are simple, their design is grounded in the pilot study in Figure 1. The more often a position is retracted, the larger its teacher–student KL, rising from 0.09 to 0.72. The more first-step mismatches a block has, the larger its KL, rising from 0.07 to 0.50. The weights therefore assign more supervision where this disagreement is larger.

Their gain is also consistent. Over uniform weighting, they improve the average in all four settings, by 0.7 points on average. This is half of HOPD's 1.4-point gain over TOPD, more than any other component (W1 & W4). They also beat confidence-based weights in all four settings, by 0.3 to 1.5 points (see our response to Reviewer E7ng). Therefore, the hindsight weights are an effective part of HOPD.

W3. Results on other dLLMs

Thank you for this suggestion. We added Fast-dLLM v2, distilling its 7B model into its 1.5B model on MATH with the SDAR-1.7B recipe, and evaluated all models. We report the results on MATH500/GSM8K with static/dynamic, and Tok./step is measured under dynamic decoding on MATH500/GSM8K.

ModelMethodMATH500GSM8KAvg.Tok./step (dyn.)
Fast-dLLM v2-1.5BBase36.8/34.367.2/64.252.0/49.22.57/2.37
SFT34.2/28.763.2/60.648.7/44.62.88/2.46
Off-policy37.4/31.166.1/61.851.8/46.43.31/2.94
TOPD40.2/34.568.7/66.154.5/50.32.12/2.03
TOPD (full-vocab)40.2/35.568.4/65.854.3/50.62.86/2.63
HOPD41.2/36.768.5/66.654.9/51.62.99/2.63
Fast-dLLM v2-7BTeacher61.4/55.386.7/84.874.0/70.12.61/2.61

HOPD achieves the best dynamic accuracy among all students, 1.0 above full-vocab TOPD and 1.3 above TOPD, and commits about 40% more tokens per step than TOPD on MATH500, 2.99 vs. 2.12.

Moreover, on LLaDA, the 0.2-point gain is only the dynamic average. The static gain is 0.5, and at the final round without checkpoint selection it is 1.1. Therefore, HOPD remains effective beyond SDAR.

Reviewer_c6CVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Hesitation-Aware On-Policy Distillation (HOPD) for diffusion language models. Its central observation is that a diffusion model predicts tokens at every masked position, while the decoder commits only some predictions; existing trace-based distillation supervises the committed positions and discards the remaining predictions, termed “hesitations.” HOPD extends teacher–student distribution matching to all masked positions along student-generated decoding trajectories. It additionally weights supervision using hindsight: positions receive greater weight when their intermediate proposals frequently disagree with the eventual token, and blocks receive greater weight when fewer initial proposals survive. The main experiments distill SDAR-1.7B and SDAR-4B from TraDo-8B-Instruct, with separate mathematical and coding training, and evaluate five benchmarks under static and dynamic decoding. The paper reports improved average accuracy, faster dynamic decoding, and a smaller extension to LLaDA-8B.

Critical strengths

The strongest contribution is the clear distinction between which predictions the decoder commits and which predictions should receive supervision. This is a useful practical insight: a prediction can provide a valuable training opportunity even when it does not immediately affect the generated state. The pilot study offers concrete motivation, showing that hesitations constitute 24.2% of recorded state–position pairs but account for 66.1% of teacher–student divergence. Appendix A also handles uncertainty appropriately by bootstrapping entire prompt trajectories rather than treating dependent token positions as independent observations. The proposed objective is coherent, and hindsight determines supervision weights rather than turning the student’s potentially incorrect final output into a hard target. Implementation simplicity is another advantage: for a fixed rollout, the expanded supervision requires no additional model forward passes or decoder modifications. The evaluation covers two student sizes, mathematical and coding tasks, divergence and weighting ablations, and an additional model family. Importantly, Appendix B.5 includes actual wall-clock throughput measurements, strengthening the efficiency evidence beyond a tokens-per-step proxy. The authors also deserve credit for including the full-vocabulary TOPD control in Appendix B.6.

Critical weaknesses

My main concern is that the evidence does not yet establish how much improvement comes specifically from supervising hesitations, or how reliably the proposed weighting helps. The headline comparison changes both the supervised position set and the divergence estimator. Although Table 10 addresses the latter difference, it substantially weakens the apparent advantage on SDAR-4B under dynamic decoding: full-vocabulary TOPD achieves a five-benchmark average of 42.2, compared with HOPD’s 42.5, leaving 0.3 percentage points rather than the headline 1.6-point gain over sampled-token TOPD. Moreover, combining the uniform all-masked results in Tables 5 and 15 gives approximately 42.0 in this setting, slightly below full-vocabulary committed-only TOPD. This does not invalidate HOPD, but it means that expanding supervision alone does not consistently demonstrate the claimed benefit. The weighting evidence is also mixed: on SDAR-4B dynamic mathematics, combined weighting improves the average only from 57.5 to 57.6; its larger static improvement is driven entirely by AIME2024 while MATH500 and GSM8K decline. On SDAR-1.7B dynamic coding, position-only weighting outperforms the combined scheme, 9.9 versus 9.4. The stronger 1.7B improvements over full-vocabulary TOPD remain meaningful positive evidence, but repeated training runs and benchmark confidence intervals are needed to assess robustness. Averaging several generated answers does not measure training-seed variability. Furthermore, Appendix B.4 selects checkpoints using MATH500 and coding benchmarks that subsequently contribute to the reported averages. This disclosed, shared protocol does not establish an unfair relative comparison, but it introduces selection optimism and weakens their interpretation as independent test results. Finally, the pilot’s association between hesitation and divergence motivates the method without proving that the particular hindsight weights identify the most useful learning signal.

Questions for authors

Several implementation details need clarification. Table 9 specifies SDAR vocabulary top-k settings of 1 for static decoding and unrestricted sampling for dynamic decoding, so these evaluations apparently differ in token sampling as well as commitment rules. The authors should clarify the sampler’s semantics and explain how repeated avg@3/avg@20 evaluation operates under top-k = 1. Table 11 also uses top-k reverse KL with k = 10 for LLaDA, whereas the main method emphasizes full-vocabulary reverse KL; the selected vocabulary set, renormalization, and treatment of excluded probability mass should be defined. The LLaDA results support applicability, but their small average gains—0.5 points static and 0.2 dynamic—and mixed task-level outcomes warrant restrained generalization claims.

4 comments
AuthorsOpenReview ↗

Response to Reviewer c6CV (1/2)

We thank the reviewer for the careful and constructive review. We address each point below.

W1. Contributions of hesitation supervision and hindsight weights

Thank you for this careful analysis. Both components of HOPD contribute, and our ablations quantify each of them. All variants except sampled-token TOPD use the full-vocabulary reverse KL, and each column adds one component. Entries are the five-benchmark average, static/dynamic, from Tables 5, 10, 13 and 15.

ModelTOPD (sampled-token)TOPD (full-vocab)+ hesitation targets+ hindsight weights (HOPD)
SDAR-4B43.9/40.943.8/42.244.1/42.044.8/42.5
SDAR-1.7B33.7/30.733.6/30.934.5/31.635.2/32.4
  • Hesitation supervision improves the average in three of the four settings, by 0.4 points on average. The exception is SDAR-4B with dynamic decoding, which we discuss in W2.
  • Hindsight weights improve it further in all four settings, by 0.7 points on average.

As the reviewer notes, on SDAR-4B with dynamic decoding most of the gain, 1.3 of 1.6 points, comes from the objective. However, this is the only such setting. Averaged over the four settings, HOPD improves sampled-token TOPD by 1.4 points: 0.3 from the objective, 0.4 from hesitation supervision and 0.7 from the hindsight weights. Therefore, both components contribute, and together they account for about three quarters of the gain.

W2. Expanding supervision alone

Expanding supervision with uniform weights improves full-vocabulary TOPD in three of the four settings, but not on SDAR-4B with dynamic decoding, 42.0 vs. 42.2. The added hesitation pairs provide more supervision signal, but they also carry most of the teacher–student divergence, 66% in our pilot (Section 3). Under uniform weights, they therefore dominate the loss and leave a smaller share of the update to the committed positions, which determine the generated tokens. This is why HOPD does not treat all supervised pairs equally, but allocates the supervision with the hindsight weights (Lines 247–250). With these weights, the average rises by 0.5 to 0.8 points over uniform weighting in all four settings. Therefore, the expanded supervision becomes a consistent gain once it is combined with the hindsight weights.

W3. Consistency of the hindsight weights

Thank you for examining the weighting ablation in detail. We agree that the size of the gain varies across benchmarks and settings, including in the cases the reviewer points out. However, its direction is consistent. The table summarizes the math three-benchmark and code two-benchmark averages from Tables 5, 13 and 15. Entries are static/dynamic.

ModelTaskUniformw_j onlyw_b onlyw_j and w_b (HOPD)
SDAR-4BMath59.3/57.558.4/56.958.4/57.660.3/57.6
SDAR-4BCode21.4/18.821.3/19.320.0/19.421.5/19.8
SDAR-1.7BMath51.0/46.749.0/46.250.6/46.551.4/47.8
SDAR-1.7BCode9.9/8.910.4/9.99.3/8.311.0/9.4

The combined weights outperform uniform weighting in all eight task-level comparisons, and in all four five-benchmark averages by 0.5 to 0.8 points. Neither weight achieves this alone. w_j alone improves over uniform weighting in three of the eight comparisons, and w_b alone in two. The two weights measure disagreement at two levels, positions and blocks, and only together do they help consistently. Therefore, we adopt the combination as the default design of HOPD.

W4. Repeated training runs and confidence intervals

Thank you for this suggestion. We trained sampled-token TOPD, full-vocabulary TOPD and HOPD three times on SDAR-4B and SDAR-1.7B, for both math and code, with the hyperparameters and checkpoint-selection rule of the paper. The table reports the five-benchmark average as the mean ± standard deviation over the three runs.

ModelDecodingTOPD (sampled-token)TOPD (full-vocab)HOPDHOPD − TOPD (sampled-token)HOPD − TOPD (full-vocab)
SDAR-4BStatic43.1 ± 0.443.0 ± 1.044.0 ± 0.7+0.9+1.0
Dynamic41.0 ± 0.441.9 ± 0.442.3 ± 0.5+1.4+0.4
SDAR-1.7BStatic33.7 ± 0.333.5 ± 0.434.2 ± 0.8+0.5+0.7
Dynamic30.2 ± 0.430.4 ± 0.331.9 ± 0.4+1.7+1.5

HOPD's mean is higher than that of both TOPD variants in all four model–decoding settings, by 0.5 to 1.7 points over sampled-token TOPD and by 0.4 to 1.5 points over full-vocabulary TOPD. Under dynamic decoding, the gains over sampled-token TOPD are 1.4 and 1.7 points, and their 95% confidence intervals, [+0.4, +2.4] and [+0.8, +2.6], exclude zero. Therefore, HOPD's gain holds across training runs. We will add this table to the appendix.

AuthorsOpenReview ↗

Response to Reviewer c6CV (2/2)

W5. Checkpoint selection

Thank you for raising this point, and for noting that the selection rule is shared by all methods. To obtain results that involve no selection at all, we evaluated every TOPD and HOPD model at round 30, the last round of the training budget fixed in Appendix B.1. Entries are static/dynamic.

ModelMethodMATH500AIME24GSM8KLCB-v2LiveBenchAvg.
SDAR-4BTOPD73.7/71.313.3/9.091.2/89.921.9/17.719.8/17.244.0/41.0
HOPD73.9/71.412.3/7.891.4/90.420.4/18.322.1/18.844.0/41.3
SDAR-1.7BTOPD62.3/57.14.2/2.381.7/79.19.3/7.29.4/6.533.4/30.4
HOPD63.0/58.11.0/6.083.6/79.811.1/8.910.9/9.933.9/32.6
LLaDA-8BTOPD37.0/37.30.2/0.083.5/83.27.5/6.76.8/6.227.0/26.7
HOPD39.0/38.10.2/0.282.2/82.48.0/6.310.9/7.028.1/26.8

Without selection, HOPD is never below TOPD in the six model–decoding settings. The gain is largest on SDAR-1.7B, +0.5/+2.2, and on LLaDA-8B under static decoding, +1.1. On SDAR-4B, the two methods reach comparable accuracy, +0.0/+0.3, but HOPD decodes faster. The table below gives the tokens committed per denoising step under dynamic decoding.

ModelMethodMATH500AIME24GSM8K
SDAR-4BTOPD2.231.891.97
HOPD2.462.092.22
SDAR-1.7BTOPD1.951.601.76
HOPD2.352.022.24

HOPD commits 10–12% more tokens per step on SDAR-4B and 20–28% more on SDAR-1.7B. Therefore, HOPD's advantage does not depend on checkpoint selection. It is more accurate on SDAR-1.7B and LLaDA-8B, and on SDAR-4B it reaches comparable accuracy while decoding faster. We have added both tables to a new Appendix C.4 and referenced it from the checkpoint-selection rule in Appendix B.4.

W6. From the pilot study to the hindsight weights

The pilot study (Figure 1) shows that the teacher–student KL grows with the number of retractions at a position and with the first-step mismatches in a block, and the hindsight weights follow these two trends. To test whether they provide a more useful learning signal, we kept everything else unchanged and compared them with uniform weighting and with confidence-based weights 1 − c, where c is the student's confidence in its proposed token. Entries are the five-benchmark average, static/dynamic.

ModelUniformConfidence (1 − c)Hindsight (HOPD)
SDAR-4B44.1/42.043.8/42.244.8/42.5
SDAR-1.7B34.5/31.633.7/31.535.2/32.4

The hindsight weights outperform both alternatives in all four settings, while confidence weighting brings no consistent gain over uniform weighting. Confidence is measured at a single step and cannot tell whether a proposal will later be kept or overturned, while the hindsight weights use the completed trajectory. Therefore, the hindsight weights provide a more useful learning signal than these alternatives.

Q1. Static decoding with top-k = 1

Following the TraceRL evaluation protocol, top-k on SDAR truncates the vocabulary, so static decoding with k = 1 is greedy, and avg@k under it is effectively a single greedy evaluation. This setting is shared by all methods, so the comparisons remain fair, and HOPD's most consistent gains are under dynamic decoding, which samples at temperature 1.0.

Q2. Top-k reverse KL on LLaDA

In Table 11, both TOPD and HOPD use the top-k reverse KL (k = 10), following the LLaDA setting of TOPD for a fair comparison. Specifically, at each position, the support is the teacher’s top-10 tokens. The teacher and student distributions are each renormalized on this support before computing the reverse KL, and the mass outside it is discarded. We have added the formal definition to Appendix C.1.

Q3. Claims on LLaDA

We agree that the gains on LLaDA are vary across tasks, but their direction is consistent: HOPD's five-benchmark average is higher than TOPD's both at the selected checkpoints, +0.5/+0.2 (static/dynamic), and at the final round without selection, +1.1/+0.1 (W5). We also added Fast-dLLM v2, distilling its 7B model into its 1.5B model on MATH.

ModelMethodMATH500GSM8KAvg.Tok./step (dyn.)
Fast-dLLM v2-1.5BBase36.8/34.367.2/64.252.0/49.22.57/2.37
SFT34.2/28.763.2/60.648.7/44.62.88/2.46
Off-policy37.4/31.166.1/61.851.8/46.43.31/2.94
TOPD40.2/34.568.7/66.154.5/50.32.12/2.03
TOPD (full-vocab)40.2/35.568.4/65.854.3/50.62.86/2.63
HOPD41.2/36.768.5/66.654.9/51.62.99/2.63
Fast-dLLM v2-7BTeacher61.4/55.386.7/84.874.0/70.12.61/2.61

HOPD has the highest average among all students under both decoding modes, +0.4/+1.3 over TOPD. Therefore, HOPD improves over TOPD on all three model families, SDAR, LLaDA and Fast-dLLM v2, which supports its generality.

Reviewer_c6CVOpenReview ↗

acknowledgment of the author response

I thank the authors for their thorough rebuttal and the substantial additional experiment. I also acknowledge that combined hindsight weighting improves over uniform weighting in all eight reported domain-level aggregates. I suggest describing W2’s loss-dominance explanation as a hypothesis and the W1 component percentages as a sequential ablation decomposition.

AuthorsOpenReview ↗

Response to Reviewer c6CV's Acknowledgment

Thank you for the positive feedback. We have revised the paper following both suggestions. In Appendix B.6, we added the uniform-weight variant to Table 10 and now describe the component gains as a sequential ablation decomposition. In Section 6, we now present the loss-dominance explanation, as motivated by the pilot study.

We hope these revisions, together with our earlier responses, have addressed your concerns, and we would be grateful if you would consider updating your evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We thank the AC and all reviewers for their time and constructive feedback. We are glad that reviewers find HOPD well motivated and simple to implement without extra forward passes (c6CV, dWMj, E7ng), effective in improving both accuracy and decoding speed (dWMj, E7ng), evaluated on math and coding with two model sizes and two decoding rules (c6CV, E7ng), and clearly written (c6CV, dWMj, E7ng).

To address the main concerns, we added the following experiments and revisions:

  • Contribution of hesitation supervision (c6CV, dWMj). A sequential ablation shows that hesitation targets alone improve full-vocabulary TOPD in three of the four settings, hindsight weights improve over uniform weighting in all four, and HOPD is the best in all four (Appendix B.6, Table 10). Section 6 now discusses why uniform supervision is not consistently better.
  • Repeated training runs (c6CV). Over three runs, HOPD's mean is higher than both TOPD variants in all four settings (response to c6CV).
  • Checkpoint selection (c6CV, E7ng). At the last round without selection, HOPD is never below TOPD in all six settings and commits 10–28% more tokens per step on SDAR (Appendix C.4).
  • Hindsight vs. confidence-based weights (E7ng, dWMj). Hindsight weights outperform confidence-based weights in all four settings (response to E7ng).
  • More models (dWMj, c6CV). On Fast-dLLM v2, HOPD achieves the highest average among all students under both decoding rules (Appendix C.5).
  • Implementation details (c6CV). We added the definition of the top-k reverse KL used on LLaDA (Appendix C.1).

We believe these results and revisions address the reviewers' main concerns, and we thank Reviewer c6CV for acknowledging our responses and the additional experiments. Revised parts are highlighted in blue, and please see the individual responses for details.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.