acceptodds
Under review as a conference paper at ICLR 2027

Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving

Abstract

As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world serving, where request management (such as scheduling and resource allocation) relies on accurate estimation of per-request inference cost. However, common cost proxies fall short for dLLMs: output length ignores that one forward pass can unmask multiple tokens, and denoising-step count ignores the heterogeneous per-step costs. We observe that the block-autoregressive generation mechanism induces a two-dimensional execution structure over output blocks and within-block denoising steps, whereas these proxies collapse it into a scalar, discarding information essential for characterizing the cost. Motivated by this insight, we propose the Denoising Workload Surface (DWS), which preserves this two-dimensional block-step structure as a probability surface to weight the heterogeneous per-step costs. We then design a coarse-to-fine training scheme that enables a lightweight prompt-only predictor to accurately predict the complex DWS. This predictor runs efficiently even on a single CPU core, avoiding GPU contention with the serving model. Since DWS decouples request-dependent execution behavior from deployment-specific cost factors, the predictor transfers across hardware configurations without retraining. In real-world serving experiments, DWS reduces cost-prediction error by up to over scalar-based predictors, while the DWS-guided shortest-job-first scheduler reduces end-to-end latency by up to for online chatbots.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion LLM, masked dLLM, LLM serving, cost estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vs9vOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies inference-cost prediction for serving block-autoregressive diffusion language models. It introduces the Denoising Workload Surface (DWS), which represents execution probabilities over output blocks and within-block denoising steps. Combining these probabilities with deployment-specific profiled costs yields a request-cost estimate. The authors develop a prompt-only predictor using a factorized parameterization, coarse-to-fine training, and INT8 quantization for efficient CPU inference. Experiments evaluate cost-model fidelity, prediction accuracy, and downstream shortest-job-first scheduling on LLaDA2.0 models. The reported results show reduced prediction error and improved online latency and offline sample-completion performance, with additional evaluations of cross-dataset, cross-model, and cross-hardware transfer.

Critical strengths
  1. Structure-aware cost representation. The paper identifies why output length and total denoising steps are insufficient cost proxies for dLLMs. DWS captures the block–step execution structure and heterogeneous step costs, providing an interpretable basis for cost estimation.
  2. Demonstrated serving benefits. The lightweight CPU predictor improves both cost prediction and end-to-end scheduling performance in online and offline workloads. Transfer experiments further support its practical usefulness across deployment settings.
  3. Coherent design. The factorized predictor satisfies the structural constraints of the denoising process by construction, and its training loss is theoretically linked to the final cost-estimation error.
Critical weaknesses
  1. Unclear contribution of DWS. In Table 1, full DWS beats Direct Cost, but DWS without Stage A and the single-horizon variant do not. It is unclear whether Direct Cost receives the same Stage-A adaptation, so the gain cannot be attributed to the two-dimensional target versus multi-horizon training. Comparing both with the same Stage-A checkpoint and matched training budget would resolve this.
  2. Limited generality. All experiments use a single model family (LLaDA2.0), and the paper itself attributes the within-block cost variation that DWS relies on to MoE routing, noting it may be small for dense models (App. C.1, C.3).
Questions for authors

See Weakness

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your constructive suggestions! We address your concerns below.


W1. Isolating DWS's advantage. DWS still outperforms Direct Cost under matched training conditions.

Following your suggestion, we initialize the Direct Cost predictor from the same Stage A checkpoint as DWS and use matched Stage-B training budgets.

After incorporating the same Stage A, Direct Cost improves from 432.29 to 408.75 ms in cost MAE, but still underperforms DWS (355.24 ms), suggesting that our design in Stage A benefits both predictors, while DWS retains a clear accuracy advantage. Moreover, unlike Direct Cost, DWS can transfer across hardware deployments without retraining the predictor.


W2. Generality Across Model Families and Architectures. We further show DWS's effectiveness beyond LLaDA2.0 and clarify how model architecture affects its benefits.

W2 (a). Cross-family generality.

We additionally evaluate SDAR-30B-A3B-Chat-b32 from the SDAR family, which differs from LLaDA2.0 in its backbone (Qwen3 vs. Ling) and training scheme. Together, these two families cover all dLLM model families natively supported by SGLang v0.5.10.

(i) Cost-prediction accuracy. We split the 20K traces into training, validation, and test sets (8:1:1). We compare with three scalar baselines: Output Length, Total Steps, and Direct Cost (which cannot transfer across deployments without retraining).

Table 1: Cost-prediction MAE (ms, ↓) on SDAR-30B-A3B-Chat-b32. | Output Length | Total Steps | Direct Cost (untransferable) | DWS (ours) | | ----: | ----: | ----: | ----: | | 1406.16 | 1145.08 | 942.77 | 760.53 |

DWS achieves 1.24× lower prediction MAE than Direct Cost, the strongest scalar baseline.

(ii) SJF-scheduler benefit. Finally, we examine whether these improvements translate into online serving gains, reporting average end-to-end (E2E) latency and time-to-first-token (TTFT).

Table 2: Online serving performance on SDAR-30B-A3B-Chat-b32. | RPS | E2E latency (s, ↓) | E2E latency (s, ↓) | TTFT (s, ↓) | TTFT (s, ↓) | | :---: | :---: | :---: | :---: | :---: | | | FCFS | DWS-SJF (ours) | FCFS | DWS-SJF (ours) | | 6 | 84.86 | 61.55 | 59.75 | 39.53 | | 8 | 139.42 | 90.51 | 113.94 | 67.86 | | 10 | 196.89 | 115.70 | 170.83 | 93.06 |

DWS-SJF consistently outperforms FCFS, reducing average E2E latency and TTFT by up to 1.70× and 1.84×, respectively.

Overall, these results suggest the generality of DWS across different dLLM model families, spanning cost prediction and serving.

W2 (b). Effect of model architecture.

DWS applies to both MoE and dense dLLMs, with greater expected benefits for the more prevalent MoE architectures. For dense models, although within-block cost variation may be smaller, DWS can still capture context-dependent cost differences across output blocks.

Notably, as discussed in Appendix C, MoE architectures dominate mainstream dLLM serving models (9/10 variants natively supported by the surveyed SGLang and vLLM releases).


We will incorporate the above results and analyses in the revision. Thank you for your time and thoughtful feedback!

Reviewer_Gs8EOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary
  1. The paper introduces the Denoising Workload Surface (DWS), a two-dimensional representation of block-level and step-level execution activity for predicting diffusion LLM inference cost.
  2. It develops a prompt-only predictor with a factorized probabilistic parameterization, coarse-to-fine training, and INT8 quantization for efficient CPU inference.
  3. DWS guides shortest-job-first scheduling, achieving up to 2.50× lower cost-prediction MAE and 1.92× lower average end-to-end latency than the reported baselines.
Critical strengths
  1. Well-motivated problem. The paper empirically demonstrates why output length and total denoising steps are insufficient cost proxies, providing a clear motivation for modeling heterogeneous per-step costs.
  2. Practical design. Separating execution activity from deployment-specific costs enables predictor reuse through re-profiling. The structured parameterization and CPU-efficient implementation support this design effectively.
  3. Substantial empirical validation. The evaluation includes prediction accuracy, training ablations, online and offline serving, cross-dataset and cross-deployment transfer, and comparisons with a resource-matched baseline and Oracle-SJF.
Critical weaknesses
  1. Limited validation of batching effects. The cost model is calibrated using individually served requests, while deployment uses batched inference. Although scheduling gains are demonstrated at batch size 96, sensitivity to batch size, batch composition, and dynamic batching remains insufficiently explored.
  2. Restricted generalization evidence. Experiments cover two models within the LLaDA2.0 family, primarily with a fixed decoding configuration and a 1,024-token output cap. Moreover, the predictor retains only the last 512 prompt tokens; long-context cost-model fidelity does not establish predictor accuracy on long-context workloads.
  3. Offline metrics favor short-request selection. Measuring time to the first 1,000 completions and completions within three minutes naturally rewards SJF, but does not establish improved total-workload throughput or representative data collection. Full-workload makespan and the resulting sample distribution should also be evaluated.
Questions for authors

Please see the weakness

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for the valuable comments! We address your concerns below.


W1. Batching effects. In short, DWS consistently improves scheduling performance under varying batching conditions.

(i) Batch size and composition. We additionally evaluate online serving at RPS=5 while varying the maximum batch size. The results below report average end-to-end (E2E) latency (s, ↓) and time-to-first-token (TTFT) (s, ↓).

Table 1: Online serving performance under varying batch sizes on LLaDA2.0-mini with LMSYS-Chat-1M workload (RPS=5). Lower is better.

Batch sizeE2E latencyE2E latencyTTFTTTFT
FCFSDWS (ours)FCFSDWS (ours)
161227.88739.321212.53724.26
32661.16394.63641.98375.73
64315.45182.04291.30158.34
96203.77118.91174.5790.62

DWS-SJF consistently achieves lower E2E latency and TTFT than FCFS across all evaluated batch sizes, where the batch composition is automatically determined by the SJF scheduler based on request priorities.

(ii) Dynamic batching. Our experiments already use the block-level dynamic batching supported by SGLang v0.5.10, where pending requests are dynamically admitted into the running batch once execution capacity becomes available, without waiting for the entire batch to complete.


W2. Generalizability. We show that DWS remains effective across different dLLM model families and decoding configurations, and further investigate the sensitivity of DWS prediction to prompt truncation.

W2(a). Generality across model families.

We additionally evaluate SDAR-30B-A3B-Chat-b32 from the SDAR family, which differs from LLaDA2.0 in its backbone (Qwen3 vs. Ling) and training scheme. Together, these two families cover all dLLM model families natively supported by SGLang v0.5.10.

(i) Cost-prediction accuracy. We split 20K traces into training, validation, and test sets at 8:1:1. We compare with three scalar baselines: Output Length, Total Steps, and Direct Cost (which cannot transfer across deployments without retraining).

Table 2: Cost-prediction MAE (ms, ↓) on SDAR-30B-A3B-Chat-b32. Output Length|Total Steps|Direct Cost (untransferable)|DWS (ours) --:|--:|--:|--: 1406.16|1145.08|942.77|760.53

DWS achieves 1.24× lower prediction MAE than Direct Cost, the strongest scalar baseline.

(ii) SJF-scheduler benefit. Finally, we evaluate online serving performance in terms of average E2E latency and TTFT.

Table 3: Online serving performance on SDAR-30B-A3B-Chat-b32. RPS|E2E latency|E2E latency|TTFT|TTFT :-:|:-:|:-:|:-:|:-: ||FCFS|DWS-SJF (ours)|FCFS|DWS-SJF (ours) 6|84.86|61.55|59.75|39.53 8|139.42|90.51|113.94|67.86 10|196.89|115.70|170.83|93.06

DWS-SJF consistently outperforms FCFS, reducing average E2E latency and TTFT by up to 1.70× and 1.84×, respectively.

W2(b). Generality across decoding configurations.

We vary the decoding threshold, a key parameter governing denoising behavior, from the default 0.95 to 0.80, 0.90, and 0.98. For each threshold, we collect 20K execution traces while keeping all other settings unchanged.

(i) Cost-prediction accuracy.

Table 4: Cost-prediction MAE (ms, ↓) across decoding thresholds. Threshold|Output Length|Total Steps|Direct Cost (untransferable)|DWS (ours) :-:|:-:|:-:|:-:|:-: 0.80|848.69|660.30|616.03|541.58 0.90|896.35|720.23|650.92|580.79 0.98|952.66|796.18|707.24|642.63

DWS consistently achieves the lowest prediction MAE, outperforming all baselines.

(ii) SJF-scheduler benefit.

Table 5: Online serving performance across RPS and decoding threshold. RPS|Threshold|E2E latency|E2E latency|TTFT|TTFT :-:|:-:|:-:|:-:|:-:|:-: |||FCFS|DWS-SJF (ours)|FCFS|DWS-SJF (ours) 6|0.8|307.16|183.36|277.24|155.25 6|0.9|303.85|186.78|274.60|158.38 6|0.98|302.39|183.85|273.15|155.42 8|0.8|508.16|298.24|478.25|270.15 8|0.9|496.83|303.63|467.49|275.22 8|0.98|497.12|302.83|467.78|274.25

DWS-SJF consistently outperforms FCFS, reducing average E2E latency and TTFT by up to 1.70× and 1.79×, respectively.

Overall, these results suggest the generality of DWS across different dLLM families and decoding configurations, spanning cost prediction and serving.

W2(c). Effect of prompt truncation. For CPU-efficient prediction, we use a lightweight encoder with a 512-token context limit, necessitating truncation for longer prompts. Following prior work [1], we retain the last 512 tokens. Moreover, the predictor also receives the original prompt length as an explicit feature.

Table 6: Cost-prediction accuracy on prompts longer than 512 tokens.

PredictorOutput LengthTotal StepsDirect CostDWS (ours)
Cost MAE (ms, ↓)1190.34736.71618.81494.31

Although MAE is higher on the >512-token subset, DWS still outperforms other predictors.

Moreover, DWS supports different output-length limits, but changing the limit requires retraining the predictor, like changing other configurations.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3. Offline evaluation metrics. Following prior studies [2, 3], we adopt the SDG scenario to evaluate whether DWS can accurately predict request inference costs and thereby guide SJF scheduling. We additionally report completion times at different milestones below.

Table 9: Time (s) to reach different completion milestones on LLaDA2.0-mini with LMSYS-Chat-1M.

Strategy1.5K (P50)2.7K (P90)3K
FCFS4948931005
DWS-SJF229796968

Since SJF only changes the serving order, it aims to mitigate head-of-line (HoL) blocking and minimize the time to complete a target number of requests, rather than the full-workload makespan. Accordingly, DWS-SJF completes the first 1.5K requests 2.16× faster than FCFS (229 vs. 494 s), while maintaining a comparable full-workload makespan (968 vs. 1005 s).

Regarding sample distribution, the first 1.5K completed requests under DWS-SJF have a lower average inference cost than under FCFS (1.12 vs. 2.73 s), indicating that DWS-guided SJF can accurately identify and prioritize lower-cost requests.


We will incorporate the above results in the revision. Thank you for your time and thoughtful feedback!


[1] Qiu et al., “Efficient Interactive LLM Serving with Proxy Model-Based Sequence Length Prediction,” AIOps @ ASPLOS, 2024.

[2] Fu et al., “Efficient LLM Scheduling by Learning to Rank,” NeurIPS 2024.

[3] Zheng et al., “Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions,” ICML 2026.

Thanks for the response. I will keep my score.

Reviewer_fZewOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Denoising Workload Surface (DWS) to predict the inference cost of diffusion language model. The authors argue that common cost proxies, such as output length and total denoising steps, do not accurately capture dLLM inference cost because different denoising steps can have different computational costs. DWS models the workload using a two-dimensional structure over output blocks and denoising steps, combined with profiled execution costs. They also trained a lightweight predictor to predict the denoising behavior of the incoming requests, which is combined with the cost model to compute the inference cost of the request. The method is evaluated on LLaDA2.0-mini and LLaDA2.0-flash using workloads from LMSYS-Chat-1M, ShareGPT, and Alpaca. The results show lower cost-prediction error than scalar-based methods and reduced serving latency when DWS is used for shortest-job-first scheduling.

Critical strengths

The paper addresses a practical problem in dLLM serving. The authors did sufficient profiling to show that inference cost can vary significantly for requests with similar output lengths or total denoising steps. The proposed cost model considers prefill, denoising, KV refresh, and cost differences across denoising steps. The decoupled design of the cost model and the predictor also reasonably separates the profiling-dependent part of the estimation from the learnable part of the estimation.

The experimental evaluation is comprehensive. The authors separately evaluate the cost model and predictor, and compare with several cost-prediction and scheduling baselines. They also did ablation studies to justify their training design. The authors also evaluate their method on workloads and a model not used for predictor training to demonstrate transferability of the proposed method.

Critical weaknesses

Please identify one or two critical weaknesses of the paper that support rejection. For each, explain briefly why it is consequential to your overall assessment. The goal is not to enumerate every negative point, but to identify and substantiate the considerations that most strongly determine your recommendation. Enter “None” if no material weakness can be identified.

The evaluation is limited to two models from the LLaDA2.0 family, using the same decoding strategy. The generalizability of the method is still not very clear. For example, there are different decoding methods for dLLMs, which could lead to different denoising behaviors. Although the authors mention that the predictor can be retrained when the decoding configuration changes, they did not evaluate the method under different decoding settings. It is unclear how well DWS would work with other decoding methods, or whether the same cost model would still be accurate.

Questions for authors

See weakness

2 comments

Thank you for the constructive comments!


We conduct additional experiments to show that DWS generalizes across both decoding configurations and dLLM families.

Q1. Robustness to decoding configurations.

The decoding threshold is a key configuration parameter that controls how aggressively masked tokens are revealed, thereby inducing different denoising behaviors. Besides the default threshold of 0.95, we evaluate three other commonly used thresholds (0.80, 0.90, and 0.98), collecting 20K execution traces and retraining the predictor separately for each threshold, while keeping all other settings unchanged.

(i) Cost-prediction accuracy. For each threshold, we split the 20K traces into training, validation, and test sets at 8:1:1. We further include Direct Cost, a scalar baseline that cannot transfer across deployments without retraining.

Table 1: Cost-prediction MAE (ms, ↓) across decoding thresholds. | Threshold | Output Length | Total Steps | Direct Cost (untransferable) | DWS (ours) | | :---: | :---: | :---: | :---: | :---: | | 0.80 | 848.69 | 660.30 | 616.03 | 541.58 | | 0.90 | 896.35 | 720.23 | 650.92 | 580.79 | | 0.98 | 952.66 | 796.18 | 707.24 | 642.63 |

DWS consistently achieves the lowest prediction MAE, outperforming Direct Cost, the strongest scalar baseline.

(ii) SJF-scheduler benefit. Finally, we evaluate online serving performance in terms of average end-to-end (E2E) latency and time-to-first-token (TTFT).

Table 2: Online serving performance across RPS and decoding threshold. | RPS | Threshold | E2E latency (s, ↓) | E2E latency (s, ↓) | TTFT (s, ↓) | TTFT (s, ↓) | | :---: | :---: | :---: | :---: | :---: | :---: | | | | FCFS | DWS-SJF (ours) | FCFS | DWS-SJF (ours) | | 6 | 0.8 | 307.16 | 183.36 | 277.24 | 155.25 | | 6 | 0.9 | 303.85 | 186.78 | 274.60 | 158.38 | | 6 | 0.98 | 302.39 | 183.85 | 273.15 | 155.42 | | 8 | 0.8 | 508.16 | 298.24 | 478.25 | 270.15 | | 8 | 0.9 | 496.83 | 303.63 | 467.49 | 275.22 | | 8 | 0.98 | 497.12 | 302.83 | 467.78 | 274.25 |

DWS-SJF consistently outperforms FCFS, reducing average E2E latency and TTFT by up to 1.70× and 1.79×, respectively.

Q2. Generality across model families.

We additionally evaluate SDAR-30B-A3B-Chat-b32 from the SDAR family, which differs from LLaDA2.0 in its backbone (Qwen3 vs. Ling) and training scheme. Together, these two families cover all dLLM model families natively supported by SGLang v0.5.10. We retrain the DWS predictor and recalibrate the cost model for SDAR.

(i) Cost-prediction accuracy.

Table 3: Cost-prediction MAE (ms, ↓) on SDAR-30B-A3B-Chat-b32. | Output Length | Total Steps | Direct Cost (untransferable) | DWS (ours) | | ----: | ----: | ----: | ----: | | 1406.16 | 1145.08 | 942.77 | 760.53 |

DWS achieves 1.24× lower prediction MAE than Direct Cost, the strongest scalar baseline, demonstrating its effectiveness on a distinct dLLM family.

(ii) SJF-scheduler benefit.

Table 4: Online serving performance on SDAR-30B-A3B-Chat-b32. | RPS | E2E latency (s, ↓) | E2E latency (s, ↓) | TTFT (s, ↓) | TTFT (s, ↓) | | :---: | :---: | :---: | :---: | :---: | | | FCFS | DWS-SJF (ours) | FCFS | DWS-SJF (ours) | | 4 | 37.80 | 39.53 | 14.52 | 15.46 | | 6 | 84.86 | 61.55 | 59.75 | 39.53 | | 8 | 139.42 | 90.51 | 113.94 | 67.86 | | 10 | 196.89 | 115.70 | 170.83 | 93.06 |

Under higher loads (RPS ≥ 6), DWS-SJF consistently outperforms FCFS, reducing average E2E latency and TTFT by up to 1.70× and 1.84×, respectively.

Overall, these results suggest the generality of DWS across different dLLM model families and decoding configurations, spanning cost prediction and serving.


We will incorporate the above results in the revision. Thank you for your time and thoughtful feedback!

Thank you for your response. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.