acceptodds
Under review as a conference paper at ICLR 2027

RETHINKING WORKSPACE CONTROL FOR OPEN-ENDED DIFFUSION LANGUAGE MODELS

Abstract

Masked diffusion language models (dLLMs) generate text by resolving masked positions in parallel, but open-ended generation remains challenging because the final output length is unknown in advance. Existing methods typically rely on model-derived signals such as EOS statistics, confidence, or prediction density to dynamically adjust the workspace, introducing additional estimation and control complexity. We find that, with the decoding state fixed and these control signals bypassed, directly changing future [MASK] exposure can substantially alter current decoding decisions. This suggests a simpler principle: rather than repeatedly estimating workspace demand from model outputs, future workspace can be organized directly around decoding progress. We therefore introduce MaskHorizon, a framework for open-ended dLLM decoding. MaskHorizon maintains a local horizon of future [MASK] positions that advances with block-level decoding progress. Each forward pass predicts over the workspace up to this horizon, while commitment remains restricted to the active block. Experiments on LLaDA and Dream across reasoning and code-generation tasks show that MaskHorizon substantially reduces decoding computation while retaining competitive generation quality. The code for MaskHorizon is available at https://anonymous.4open.science/r/MaskHorizon/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, efficient decoding, variable-length generation, masked diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_avvKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MaskHorizon, a simple training-free framework for open-ended diffusion language model decoding that grows the workspace block by block while keeping a local reserve of future masked positions as context. The method avoids relying on model-derived signals such as EOS or confidence for workspace growth and achieves lower decoding cost across multiple tasks and two DLM backbones.

However, the main evaluation uses only 100 examples for several benchmarks, which is too limited to support reliable accuracy conclusions, especially when many reported quality differences are only a few percentage points.

Critical strengths

(1) The idea of this work is interesting. It does not require signals such as EOS or confidence to adjust the inference length of DLMs. Instead, it performs inference block by block in a way similar to AR generation, while keeping a certain amount of consecutive masks as context.

(2) The proposed method is simple and clear.

Critical weaknesses

(1) The evaluation uses only 100 samples, which is too few. Considering that this is a training-free inference method, and the proposed method itself accelerates inference, the evaluation should not require excessive computational resources. However, the main evaluation only uses 100 samples from the test sets (e.g., the full GSM8K test set contains 1,319 samples and MBPP contains 500 samples). This makes the evaluation results less stable than evaluation on the full datasets and insufficient to support reliable accuracy conclusions.

(2) The metrics used to measure speedup are not sufficiently reasonable. The authors point out that different methods process different numbers of tokens at each step, and therefore NFE is not a suitable metric, but NFE is still reported in the main results. In addition, the authors introduce the CFP metric. However, the computational cost of the model is not directly proportional to the number of tokens, which is related to the O(n2)O(n^2) complexity of attention. Therefore, CFP is still not a sufficiently reasonable metric for measuring computational cost.

Questions for authors

(1) The hyper-parameter initial length N0 uses different values for different benchmarks. How were these task-specific defaults determined?

13 comments

Thank you for carefully reviewing our paper and providing specific suggestions for improvement. We particularly appreciate your recognition of the value of our approach: rather than relying on model signals such as EOS probability or confidence to determine when the workspace grows, it advances block by block with decoding progress while retaining contiguous future MASK tokens as context. We also appreciate your positive assessment of the simplicity and clarity of the method.

W1

The evaluation uses only 100 samples, which is too few. Considering that this is a training-free inference method, and the proposed method itself accelerates inference, the evaluation should not require excessive computational resources. However, the main evaluation only uses 100 samples from the test sets (e.g., the full GSM8K test set contains 1,319 samples and MBPP contains 500 samples). This makes the evaluation results less stable than evaluation on the full datasets and insufficient to support reliable accuracy conclusions.

R1:

We agree that 100 examples are insufficient to establish small accuracy differences. The original evaluation used a shared 100-example subset to compare workspace and commitment strategies for efficiency and mechanistic analysis. Similar settings appear in ATTS (MATH100) [1], QSpec (100 examples per task across six tasks) [2], and PTD (100 examples each from GSM8K and MBPP) [3]. These precedents do not replace larger-scale quality evaluation, which we therefore added in response to your suggestion.

We added expanded evaluations of LLaDA and Dream on GSM8K (1,319 examples), MATH500 (500 examples), and MBPP (up to 500 examples), rather than reporting results based only on the original subset of 100 examples. Below, we present generation quality and the number of model calls; time and estimated computational costs are listed separately in W2.

Table 1 shows that MaskHorizon achieves slightly higher scores than the listed baselines on all three tasks with LLaDA. NFE is lower than all listed baselines on GSM8K and MBPP; on MATH500, it is lower than DAEDAL and Fast-dLLM but higher than AdaBlock-dLLM. These full-scale results extend the quality–cost evaluation beyond the original 100-example subsets.

Quality and Decoding Statistics

Table 1

ModelDatasetMethodScore (%) ↑NFE ↓
LLaDAGSM8KAdaBlock-dLLM82.64118.23
DAEDAL82.94103.63
Fast-dLLM82.8793.19
MaskHorizon83.3279.83
MATH500AdaBlock-dLLM38.00159.79
DAEDAL39.40197.63
Fast-dLLM38.20185.96
MaskHorizon39.60173.18
MBPPAdaBlock-dLLM37.27147.99
DAEDAL37.47161.41
Fast-dLLM36.27139.82
MaskHorizon37.88103.61
DreamGSM8KAdaBlock-dLLM81.1293.10
DAEDAL81.8897.87
MaskHorizon82.3488.53
MATH500AdaBlock-dLLM47.60173.54
DAEDAL46.20162.99
MaskHorizon44.20123.52
MBPPAdaBlock-dLLM50.40127.65
DAEDAL50.50139.23
MaskHorizon51.00121.13

W2

The metrics used to measure speedup are not sufficiently reasonable. The authors point out that different methods process different numbers of tokens at each step, and therefore NFE is not a suitable metric, but NFE is still reported in the main results. In addition, the authors introduce the CFP metric. However, the computational cost of the model is not directly proportional to the number of tokens, which is related to the (O(n^2)) complexity of attention. Therefore, CFP is still not a sufficiently reasonable metric for measuring computational cost.

R2:

Thank you for raising this concern. We agree that neither NFE nor CFP alone fully captures computational cost. We retain NFE for comparison with prior work, but use CFP to account for an important distinction that NFE overlooks: each forward call may process a different number of tokens. This accounting follows related evaluation practices in SIDM [4], which accumulates the sequence lengths processed during inference, and WeDLM [5], which counts token instances processed across decoding forwards, including recomputation. While WeDLM excludes initial prefill, our CFP = ∑ᵢNᵢ includes the prompt in each forward input length Nᵢ.

CFP therefore complements NFE as a measure of cumulative token processing, rather than a complete measure of computation. To address your concern about quadratic attention cost, we additionally report estimated FLOPs that include this cost, alongside actual decoding time. In the revised manuscript, we will present these two measures in the main text and retain the CFP analysis and results in the appendix as supplementary evidence.

We estimate decoding FLOPs from the recorded per-step input lengths, including the prompt, and the model architecture. The estimate covers Q/K/V/O projections, the MLP, the vocabulary output projection, and both attention matrix multiplications (QKᵀ and AV), thereby accounting for quadratic attention cost. One multiply-add counts as two FLOPs. Table 2 reports mean TFLOPs per sample; ratios compare the displayed mean computational costs with MaskHorizon, not measured speedup; valid sample counts may differ.

As shown in Table 2, after including the quadratic term, the estimated computational cost reductions across the 20 baseline comparisons using the displayed means range from 16.94% to 72.41%, indicating that the computational savings of the listed configurations do not arise from ignoring the quadratic cost of attention.

Table 2

ModelDatasetAdaBlock-dLLMDAEDALFast-dLLMMaskHorizon
LLaDAGSM8K1679.78(3.62×)724.61(1.56×)1324.21(2.86×)463.51(1.00×)
MATH5002307.42(1.71×)2172.01(1.61×)2689.90(2.00×)1345.65(1.00×)
MBPP2205.94(3.35×)1709.22(2.60×)2082.96(3.17×)657.62(1.00×)
HumanEval3506.24(2.94×)3826.26(3.20×)3316.19(2.78×)1193.93(1.00×)
DreamGSM8K1229.02(2.71×)545.64(1.20×)—453.21(1.00×)
MATH5002329.34(3.02×)1164.13(1.51×)—770.21(1.00×)
MBPP1797.56(2.47×)938.32(1.29×)—726.33(1.00×)
HumanEval2095.73(2.77×)1134.82(1.50×)—756.68(1.00×)

We also provide actual decoding time to reflect runtime factors not covered by the theoretical estimates above. Table 3 reports the mean decoding time per sample in seconds, excluding model loading, warm-up, and scoring. Times come from historical runs across different periods and concurrent workloads and are descriptive, not controlled speedup measurements.

Table 3

ModelDatasetAdaBlock-dLLMDAEDALFast-dLLMMaskHorizon
LLaDAGSM8K15.927.0112.464.51
MATH50021.4920.1825.2312.49
MBPP20.8915.7619.706.33
HumanEval32.7734.5930.9411.01
DreamGSM8K10.565.47—4.44
MATH50019.9811.43—7.26
MBPP15.359.17—6.95
HumanEval17.7510.89—7.16

Q1

The hyper-parameter initial length N0 uses different values for different benchmarks. How were these task-specific defaults determined?

R3:

Thank you for your question. The initial lengths used for GSM8K, MATH500, MBPP, and HumanEval in the original manuscript are task-level empirical settings, rather than optimal values determined through a unified search.

We compared N₀=192, 224, and 256 on the full evaluation sets of all four tasks using LLaDA, fixing all other parameters and evaluation procedures. We report every tested setting and retain the original defaults without reselecting them. Table 4 reports "Score (%) / NFE." The results assess applicability within these tasks, rather than establish a universally optimal initial value.

The N₀ sensitivity results in Table 4 show that the GSM8K default of 192 maintains comparable quality with lower NFE; the MATH500 default of 256 achieves a higher score within the tested range, but at a higher cost; MBPP achieves its highest score at 224, whereas the default of 192 has a lower cost; and increasing HumanEval from the default of 224 to 256 passes only one additional problem while requiring more calls. We therefore regard the defaults as task-level empirical starting points.

Table 4

DatasetN₀=192N₀=224N₀=256
GSM8K83.32 / 79.8383.09 / 86.6483.24 / 92.55
MATH50034.00 / 150.8736.20 / 165.2639.60 / 173.18
MBPP37.88 / 103.6139.88 / 111.4238.88 / 115.35
HumanEval40.85 / 136.7843.29 / 144.9043.90 / 148.85

References

[1] Jing Xiong, Qiujiang Chen, Fanghua Ye, Zhongwei Wan, Chuanyang Zheng, Chenyang Zhao, Hui Shen, Alexander Hanbo Li, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Lingpeng Kong, and Ngai Wong. ATTS: Asynchronous Test-Time Scaling via Conformal Prediction. In International Conference on Learning Representations (ICLR), 2026.

[2] Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, and Chuan Wu. QSpec: Speculative Decoding with Complementary Quantization Schemes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4779–4795, 2025. DOI: 10.18653/v1/2025.emnlp-main.240.

[3] Zipeng Gao, Zhi Zheng, Qingrong Xia, Junda Lin, Ziwei Zhao, Tong Xu, Zhefeng Wang, and Enhong Chen. Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting. In Conference on Language Modeling (COLM), 2026.

[4] Andre He, Sean Welleck, and Daniel Fried. Reasoning with Latent Tokens in Diffusion Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2026.

[5] Aiwei Liu et al. WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference. In International Conference on Machine Learning (ICML), 2026.

Reviewer_oQCCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MaskHorizon, a training-free method for open-ended diffusion decoding. It maintains a local reserve of future masked positions, predicts over the exposed workspace, and commits tokens only within the active block. Experiments on LLaDA and Dream study workspace exposure, reserve size, commitment scope, and growth rules, reporting lower decoding cost with task-dependent quality changes.

Critical strengths
  1. The distinction between workspace exposure and token commitment is clear and useful. The matched-state intervention shows that future masks can affect current predictions, while the commitment-scope comparison supports exposing future space without immediately committing its tokens.

  2. The controlled experiments are informative. The reserve, growth-policy, and granularity studies help explain the design choices. The paper also clearly separates native baselines from adapted controllers and reports the failed Dream Fast-dLLM reproduction separately.

Critical weaknesses
  1. Independent validation is missing. Three main benchmarks use only 100 examples each, and Appendix B states that there is no validation split and that earlier threshold choices used the same test examples. The task-specific initial workspace settings make it difficult to assess how well the selected defaults generalize beyond these cohorts.

  2. The efficiency gains are not established at matched quality. On LLaDA MATH500, MaskHorizon scores 42 compared with 49 for AdaBlock-dLLM; on Dream MATH500, it scores 52 compared with 56 for DAEDAL-port. The paper acknowledges these differences, but a matched-quality comparison would make the practical efficiency advantage more convincing.

Questions for authors

N/A

6 comments

Thank you for your detailed review and your positive assessment of the mechanistic analysis and experimental transparency. We particularly appreciate your recognition of the distinction between workspace exposure and token commitment, your observation that the matched-state interventions support the effect of future MASK tokens on current predictions, and your acknowledgment that the commitment-range comparisons support the design of "exposing future space without immediately committing its tokens."

W1

Independent validation is missing. Three main benchmarks use only 100 examples each, and Appendix B states that there is no validation split and that earlier threshold choices used the same test examples. The task-specific initial workspace settings make it difficult to assess how well the selected defaults generalize beyond these cohorts.

R1:

Thank you for your suggestion. Our evaluation protocol follows the experimental settings of related inference-time decoding studies, including AdaBlock-dLLM [1] and DAEDAL [2], published at ICLR 2026, focusing on comparisons of quality and inference cost on standard benchmarks rather than constructing a separate validation set for parameter selection. Specifically, we did not set aside an additional hyperparameter validation set; instead, we used task-level empirical configurations and supplemented the evaluation with experiments using different initial workspace lengths to show the quality–efficiency trade-offs of the defaults and nearby settings. Below, we further explain the rationale for using 100 examples in the original manuscript and present the newly added full-scale evaluation results.

The original manuscript used a fixed subset of 100 examples primarily to compare workspace allocation, commitment strategies, and per-step computational costs on inputs shared across methods, providing a consistent basis for efficiency and mechanistic analyses. Similar settings also appear in inference efficiency research: ATTS [3] uses MATH100, QSpec [4] samples 100 examples from each of six datasets for acceleration evaluation, and PTD [5] uses 100 examples each from GSM8K and MBPP. However, these precedents do not replace larger-scale quality evaluation. We recognize that, with 100 examples, a small number of gains or losses can cause noticeable percentage changes, so small accuracy advantages should not be emphasized on this basis alone.

To address your concern that the earlier threshold choices used the same test examples as the evaluation, we expanded the evaluation from the original fixed subset of 100 examples to GSM8K (1,319 examples), MATH500 (500 examples), and MBPP (up to 500 examples), adding full-scale results for LLaDA and Dream to examine the performance of the listed configurations over a broader range of examples. We also supplemented this with a single-factor analysis of N₀ while holding all other parameters fixed, reporting the results for the defaults and nearby settings in full. The full-scale evaluation results in Table 1 broaden the evaluation coverage and show the quality–cost performance of the listed configurations.

Table 1

ModelDatasetMethodScore (%) ↑NFE ↓CFP (K) ↓
LLaDAGSM8KAdaBlock-dLLM82.64118.23108.54
DAEDAL82.94103.6347.50
Fast-dLLM82.8793.1985.56
MaskHorizon83.3279.8330.49
MATH500AdaBlock-dLLM38.00159.79148.98
DAEDAL39.40197.63140.90
Fast-dLLM38.20185.96173.67
MaskHorizon39.60173.1888.02
MBPPAdaBlock-dLLM37.27147.99142.30
DAEDAL37.47161.41111.02
Fast-dLLM36.27139.82134.37
MaskHorizon37.88103.6143.19
DreamGSM8KAdaBlock-dLLM81.1293.1084.72
DAEDAL81.8897.8738.15
MaskHorizon82.3488.5331.72
MATH500AdaBlock-dLLM47.60173.54160.48
DAEDAL46.20162.9981.05
MaskHorizon44.20123.5253.76
MBPPAdaBlock-dLLM50.40127.65123.50
DAEDAL50.50139.2365.43
MaskHorizon51.00121.1350.45

The initial lengths used for GSM8K, MATH500, MBPP, and HumanEval in the original manuscript are task-level empirical settings, rather than optimal values determined through a unified search.

To examine the applicability of these settings over a broader range of examples, we compared N₀=192, 224, and 256 on the full evaluation sets of all four tasks using LLaDA, holding all other parameters and evaluation procedures fixed and changing only N₀.

The N₀ sensitivity results in Table 2 show that the GSM8K default of 192 maintains comparable quality with lower NFE; the MATH500 default of 256 achieves a higher score within the tested range, but at a higher cost; MBPP achieves its highest score at 224, whereas the default of 192 has a lower cost; and increasing HumanEval from the default of 224 to 256 passes only one additional problem while requiring more calls. We therefore regard the defaults as task-level empirical starting points rather than the highest-scoring configurations for each task. These results demonstrate quality–cost trade-offs on the existing tasks, rather than implying that the same settings are optimal for all unseen tasks.

Table 2

DatasetN₀=192N₀=224N₀=256
GSM8K83.32 / 79.8383.09 / 86.6483.24 / 92.55
MATH50034.00 / 150.8736.20 / 165.2639.60 / 173.18
MBPP37.88 / 103.6139.88 / 111.4238.88 / 115.35
HumanEval40.85 / 136.7843.29 / 144.9043.90 / 148.85

W2

The efficiency gains are not established at matched quality. On LLaDA MATH500, MaskHorizon scores 42 compared with 49 for AdaBlock-dLLM; on Dream MATH500, it scores 52 compared with 56 for DAEDAL-port. The paper acknowledges these differences, but a matched-quality comparison would make the practical efficiency advantage more convincing.

R2:

Thank you for suggesting that efficiency be compared at comparable generation quality. The original manuscript used a smaller fixed subset, where a small number of gains or losses could lead to relatively noticeable percentage changes. We therefore conducted a full-scale evaluation, with the following results:

The full-scale evaluation results in Table 1 show that, on LLaDA, MaskHorizon achieves scores of 83.32%, 39.60%, and 37.88% on GSM8K, MATH500, and MBPP, respectively, all comparable to or slightly higher than those of the listed baselines, while having lower estimated computational costs. In particular, on MATH500, MaskHorizon and DAEDAL score 39.60% and 39.40%, with computational costs of 1345.65 and 2172.01 TFLOPs/sample, respectively, a reduction of approximately 38.05%. This supports cost savings at comparable observed quality, but we do not interpret the difference of 0.20 percentage points as a significant quality advantage.

Dream also shows comparable or slightly higher scores with lower estimated computational costs on GSM8K and MBPP. On MATH500, MaskHorizon scores 44.20%, below AdaBlock-dLLM at 47.60% and DAEDAL at 46.20%. We therefore interpret this as a quality–cost trade-off and do not include it in the conclusion of an advantage at comparable quality.

Table 1

ModelDatasetMethodScore (%) ↑NFE ↓CFP (K) ↓
LLaDAGSM8KAdaBlock-dLLM82.64118.23108.54
DAEDAL82.94103.6347.50
Fast-dLLM82.8793.1985.56
MaskHorizon83.3279.8330.49
MATH500AdaBlock-dLLM38.00159.79148.98
DAEDAL39.40197.63140.90
Fast-dLLM38.20185.96173.67
MaskHorizon39.60173.1888.02
MBPPAdaBlock-dLLM37.27147.99142.30
DAEDAL37.47161.41111.02
Fast-dLLM36.27139.82134.37
MaskHorizon37.88103.6143.19
DreamGSM8KAdaBlock-dLLM81.1293.1084.72
DAEDAL81.8897.8738.15
MaskHorizon82.3488.5331.72
MATH500AdaBlock-dLLM47.60173.54160.48
DAEDAL46.20162.9981.05
MaskHorizon44.20123.5253.76
MBPPAdaBlock-dLLM50.40127.65123.50
DAEDAL50.50139.2365.43
MaskHorizon51.00121.1350.45

References

[1] Guanxi Lu, Hao Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, and Hongxiang Fan. AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size. In International Conference on Learning Representations (ICLR), 2026.

[2] Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models. In International Conference on Learning Representations (ICLR), 2026.

[3] Jing Xiong, Qiujiang Chen, Fanghua Ye, Zhongwei Wan, Chuanyang Zheng, Chenyang Zhao, Hui Shen, Alexander Hanbo Li, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Lingpeng Kong, and Ngai Wong. ATTS: Asynchronous Test-Time Scaling via Conformal Prediction. In International Conference on Learning Representations (ICLR), 2026.

[4] Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, and Chuan Wu. QSpec: Speculative Decoding with Complementary Quantization Schemes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4779–4795, 2025. DOI: 10.18653/v1/2025.emnlp-main.240.

[5] Zipeng Gao, Zhi Zheng, Qingrong Xia, Junda Lin, Ziwei Zhao, Tong Xu, Zhefeng Wang, and Enhong Chen. Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting. In Conference on Language Modeling (COLM), 2026.

Reviewer_DQo1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MaskHorizon, a training-free method for open-ended generation with diffusion language models. Instead of using model signals such as EOS probability or confidence to decide when to expand the workspace, it maintains a local reserve of future MASK tokens based on decoding progress. The method also allows the model to predict over the full exposed workspace while only committing tokens in the active block. Experiments on LLaDA and Dream across reasoning and coding tasks show lower decoding costs than existing methods, with competitive generation quality.

Critical strengths
  1. The paper provides a useful insight that exposing future MASK tokens can change current decoding decisions. The matched-state experiments support this observation and motivate the proposed method.

  2. MaskHorizon is simple and training-free. It reduces the need for model-based workspace control, and experiments on two model backbones show consistent reductions in NFE and CFP compared with the tested variable-length baselines.

Critical weaknesses
  1. The evaluation covers only LLaDA and Dream. It remains unclear whether the method also works well with block-wise diffusion language models (e.g. LLaDA 2.1 or DiffusionGemma), where decoding and commitment may follow different designs.
Questions for authors

Does MaskHorizon also benefit block-wise diffusion language models, such as LLaDA 2.1 or DiffusionGemma? Since these models use block-wise decoding, it would be useful to understand whether the same strategy applies directly, or whether changes to the method are needed.

3 comments

Thank you for your careful review of our paper. We particularly appreciate your recognition of the observation that "exposing future MASK tokens affects current decoding decisions" and the support provided by the matched-state experiments for this observation. We also appreciate your positive assessment of the simple, training-free design of MaskHorizon.

Your suggestion regarding blockwise diffusion models raises an important boundary of applicability for this principle: whether future workspace remains useful when the model's original attention and commitment structures change, and what adaptations are needed. To directly address this question, we conducted supplementary experiments on LLaDA2.1-mini.

W1 & Q1

The evaluation covers only LLaDA and Dream. It remains unclear whether the method also works well with block-wise diffusion language models (e.g. LLaDA 2.1 or DiffusionGemma), where decoding and commitment may follow different designs.

Does MaskHorizon also benefit block-wise diffusion language models, such as LLaDA 2.1 or DiffusionGemma? Since these models use block-wise decoding, it would be useful to understand whether the same strategy applies directly, or whether changes to the method are needed.

R1:

MaskHorizon can be adapted to blockwise diffusion models by retaining its core design: appending future MASK blocks beyond the active block while committing tokens only within the active block. To evaluate this approach, we conducted supplementary experiments on LLaDA2.1-mini, keeping the workspace design as close as possible to our original setting. Because block-causal attention normally prevents the active block from accessing later blocks, we adjusted the attention mask to allow queries at positions being generated in the active block to attend to the appended future MASK blocks. This adaptation requires no additional training or changes to model weights.

Unlike LLaDA and Dream, which use bidirectional attention in this paper, the block-causal attention of LLaDA2.1-mini does not allow the current block to access subsequent blocks by default. Thus, simply appending MASK blocks is insufficient to implement our mechanism. We accordingly allow queries at positions to be generated in the current active block to access the appended future MASK blocks, while preserving the original attention structure for the prompt, committed historical positions, and future positions. This adaptation preserves the core design of our paper, "separating exposure of future workspace from token commitment," and adjusts only the attention visibility required to implement this mechanism under the block-causal structure.

We evaluated the method on GSM8K, MATH500, HumanEval, and MBPP, with the following results:

As shown in the blockwise-model evaluation in Table 1, on GSM8K, MaskHorizon maintains the same accuracy, while reducing NFE from 91.75 to 65.03 and the recorded mean time from 26.03 s to 18.32 s. On MBPP, the number of correct answers increases from 43 to 44, while both NFE and mean time decrease. Lower NFE and mean time are also observed on MATH500 and HumanEval, but with 2 and 1 fewer correct answers, respectively, reflecting task-dependent quality–efficiency trade-offs. These results provide preliminary evidence for the feasibility of this design principle in blockwise models.

Table 1

DatasetMethodScore (%)NFEMean Time (s)
GSM8KBaseline85.0091.7526.03
MaskHorizon85.0065.0318.32
MATH500Baseline47.00100.9228.97
MaskHorizon45.0090.4326.13
HumanEvalBaseline58.00122.3335.10
MaskHorizon57.00118.4734.08
MBPPBaseline43.00135.5338.25
MaskHorizon44.00114.2532.24

The author's responses addressed my concern. I think the insight may be potentially valuable for future research, although the scope maybe relatively small. I maintain my rating.

Thank you for your careful reading and constructive discussion. We are glad that our responses addressed your concern, and your recognition of the potential value of our insight for future research is encouraging.

As diffusion language models continue to evolve rapidly, we look forward to extending our research to newer models such as LLaDA2.1, SDAR, and DiffusionGemma. Your feedback has helped us clarify the contribution of this work and identify directions for future research.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your thoughtful reviews and the time you have devoted to the discussion. We are encouraged that all three reviewers recognized valuable aspects of our work, including the insight that future MASK positions can influence current predictions, the distinction between workspace exposure and token commitment, and the simplicity of MaskHorizon’s training-free design.

We are especially grateful for the questions about evaluation scale, computational-cost measurement, and task-specific initialization. Following additional experiments and clarifications to the main-text presentation, Reviewer avvK raised their rating to weak accept, while Reviewer DQo1 confirmed that our responses addressed their concern. Reviewer oQCC raised related questions about evaluation coverage and the applicability of task-specific settings, alongside the quality–efficiency trade-off. The expanded evaluations and shared-initialization results therefore also bear directly on these concerns. We recognize the substantial demands on reviewers’ time and, while we have not yet received further feedback from oQCC, hope this overlapping evidence will be useful to the AC in assessing the remaining questions.

During the rebuttal, we added larger-scale evaluations, estimated FLOPs and decoding-time comparisons, expanded N₀ experiments, and a shared-initialization comparison across four tasks. These results and clarifications will be integrated into the revised manuscript, with shared and task-specific settings presented side by side in the main text.

Our work offers an insight into diffusion decoding that leads to an effective workspace-control method: future MASK positions are not merely empty slots waiting to be filled; how they are exposed can shape what the model predicts now. MaskHorizon translates this observation into a training-free decoding method that separates workspace exposure from token commitment. The reviewers’ recognition and the additional experimental evidence give us greater confidence in both the value of this insight and the effectiveness of MaskHorizon. We believe the paper contributes more than a decoding technique: it identifies workspace organization as a meaningful direction for improving open-ended diffusion generation. We hope this summary helps the AC assess the paper’s contribution in light of the additional evidence and discussion.

Thank you again for the care and constructive engagement you have given our work. We wish you all the best in your research and future endeavors.

Best regards,
Authors of Submission #38200

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.