acceptodds
Under review as a conference paper at ICLR 2027

Learning Length Priors for Canvas Allocation in Masked Diffusion Language Models

Abstract

Masked diffusion language models generate responses by progressively denoising a preallocated canvas of masked tokens. A globally fixed canvas can constrain responses that need more space or waste computation on shorter ones. Recent methods address this issue by dynamically adjusting the response canvas during denoising, but the required canvas length is still unknown before denoising begins. We find that the information predictive of reference response length could be extracted from prompt representations in dLLMs before denoising begins. We propose CALP, an approach to Canvas Allocation with Learned Priors derived from frozen dLLM prompt representations. A lightweight predictor determines the canvas length before denoising, leaving the backbone and denoising procedure unchanged. We evaluate our approach on LLaDA-8B-Instruct, LLaDA-1.5, and Dream-7B-Instruct across five benchmarks covering mathematical reasoning, coding, and instruction following, against baselines with a fixed canvas and adaptive methods including DAEDAL and . Compared with DAEDAL, our method improves the average task score by 1.43 percentage points, reduces the average number of allocated response positions by 55.5%, and achieves an observed inference speedup of 3.75. Relative to -EOS, it improves the average task score by 2.21 percentage points and achieves an observed inference speedup of 1.83.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_7xbEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CALP, which predicts response length from frozen masked diffusion model representations using a lightweight MLP. The predicted length determines a fixed canvas before denoising. Experiments on three backbones and five benchmarks report improvements in quality and inference efficiency.

Critical strengths
  1. The method is simple and leaves the backbone and denoising procedure unchanged, making integration straightforward.
  2. Evaluation across three backbones and five benchmarks provides useful evidence of practical benefits, with prediction overhead included in timing.
Critical weaknesses
  1. Upfront prediction is insufficiently justified under block diffusion. Incremental block-wise extension with EOS stopping reduces the need to predict total length beforehand. Although CALP compares against dynamic methods, it does not isolate its benefit under matched execution and stopping conditions. Under-allocation can constrain responses, while over-allocation wastes computation and may impair termination.

  2. The length model is oversimplified. Response length is naturally described by a conditional distribution p(L∣x)p(L\mid x), since the same prompt can admit multiple valid response lengths. CALP reduces this uncertainty to a single log-length estimate, without accounting for tail risk or asymmetric allocation costs. Distributional or conditional quantile prediction would better support uncertainty-aware canvas allocation.

Questions for authors
  1. Does CALP-based allocation with full-canvas decoding offer a better trade-off in peak GPU memory, generation quality, and latency than block-wise generation with EOS stopping under comparable settings?

  2. How accurately does CALP model the conditional response-length distribution, particularly its long tail? Do lower length-prediction errors consistently translate into better generation quality and efficiency?

  3. Would CALP remain effective for chain-of-thought generation, where reasoning length can vary substantially for the same prompt? Would this require retraining the predictor or updating length estimates during decoding, and what fine-tuning data and procedure would be needed?

  4. Can the authors track denoising trajectories under deliberately under- and overestimated canvases, including EOS emergence, truncation, and accuracy changes, to clarify how canvas allocation affects generation beyond computational cost?

Reviewer_GDHEOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper predicts response canvas length from the last prompt token's final-layer representation. A small predictor allocates the canvas before denoising, reducing unnecessary computation. Experiments report modest average quality improvements and substantial speedups over dynamic allocation methods.

Masked dLLMs commonly allocate a fixed response canvas before generation. A large canvas wastes computation on short responses, while a small canvas can constrain longer answers. CALP runs the frozen model on the prompt alone and uses a two-layer MLP to predict reference-response length. It then allocates the predicted number of masked positions and performs standard denoising. The canvas remains fixed throughout generation.

The main configuration selects separate length predictors for math, code, and general instructions. A shared predictor is also evaluated. Only these lightweight components are trained; the backbone remains frozen.

Critical strengths

The idea is straightforward and addresses a concrete computational inefficiency. It requires no backbone retraining and is evaluated across multiple models and benchmarks. Reported inference times include the additional prompt-only forward pass.

Critical weaknesses
  • Unclear match to generation needs: The predictor estimates reference-answer length from the prompt alone. Actual generation also depends on the response canvas. The paper should show whether the predicted canvas gives the model enough space to produce a correct answer.
  • Missing randomized controls: The gains may come from choosing better canvas lengths overall. Shuffle predicted lengths within each task family to test whether matching each length to its prompt actually helps.
  • Insufficient prediction-error analysis: The reported R2R^2 does not show how often the canvas is too short or whether this hurts generation quality. Report prediction errors in tokens, underallocation rates, and their effect on correctness.
  • Unclear broader applicability: The main method uses separate predictors for task families. Its performance on unfamiliar tasks is unclear, and its benefit may be smaller for block diffusion models that can keep generating blocks until EOS.

Baselines and comparisons: The paper includes fixed-canvas and dynamic-allocation baselines. However, the comparisons lack important controls for establishing the value of prompt-specific prediction. These include shuffled predicted lengths, random allocation matched to the predicted length distribution, and simple task-family allocation policies. The main fixed-canvas comparison averages several configurations rather than evaluating a single policy selected on validation data.

Taken together, these gaps leave the central attribution and reliability claims insufficiently established. I recommend rejection in the current form.

Questions for authors

None.

Reviewer_tEckOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a canvas length prediction module for diffusion-based LLMs, so that the canvas length can be adaptive with respect to different prompts or tasks, given the trade-offs that larger canvas wastes compute while smaller canvas has limited capacity. The canvas length prediction module, named CALP, is trained once and can be applied to different families of dLLMs without further fine-tuning. Experiments show the effectiveness of CALP across various benchmarks in terms of efficiency-performance trade-offs.

Critical strengths
  1. The method shows promising results across different benchmarks and different dLLMs without specific tuning or adaptation.
Critical weaknesses
  1. My main concern is that using a deep neural network to predict an important hyperparameter seems like the only contribution of this work and adds not much new knowledge to the community.
  2. Despite in the paper, the authors show that CALP has certain generalizability, a small model is still at the risk of overfitting to its training data and impacting the generalizability of the frozen dLLMs.
Questions for authors

Please refer to the weeakness section.

Reviewer_qxYzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CALP, which predicts response length in advance from prompt representations using a lightweight length predictor and allocates the canvas accordingly in masked diffusion LLMs. The authors show that response length can be predicted reasonably well from frozen dLLM prompt representations, with simple probes achieving approximately. Across three dLLMs and five benchmarks, CALP slightly improves task performance over existing adaptive methods while substantially reducing inference time.

Critical strengths

The method is simple and clear. It achieves substantial computational gains without modifying the backbone or the denoising procedure, with an average 3.75× speedup over DAEDAL and 1.83× over ρ-EOS.

Critical weaknesses
  1. CALP predicts a single scalar response length rather than a distribution over possible lengths, and uses it as a fixed canvas throughout generation. However, the same prompt can admit valid responses of different lengths, so a single fixed length may fail to capture this variation. In fact, [1] points out that treating output length as a single point estimate can be problematic under stochastic decoding and that length uncertainty should be considered.

  2. The novelty also appears limited. Prior works [2, 3] have already proposed predicting an appropriate response length from the input and using it for diffusion generation, making CALP appear closer to a marginal extension of an existing direction.

Questions for authors
  1. Since the same prompt can induce a distribution over response lengths, why is fixing the canvas to a single deterministic length sufficient? Does the method remain effective under stochastic sampling with non-zero temperature?

  2. Compared with [2, 3], what is the main new impact of CALP to the community?

References

[1] Zheng et al., Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions, ICML, 2026.

[2] Rossi et al., Diffusion Language Models Are Natively Length-Aware, arXiv, 2026.03.

[3] Rottoli et al., Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs, IJCNN, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.