acceptodds
Under review as a conference paper at ICLR 2027

STP: Efficient Reinforcement Learning for Diffusion Large Language Models

Abstract

Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs). However, applying RL to dLLMs faces unique challenges in efficiency and stability. To address these challenges, we propose Spatio-Temporal Pruning (STP), a framework that removes redundancy along both dimensions of the diffusion process. STP employs static priors, anchors reliable positions, and concentrates exploration, while leveraging bidirectional masked generation to bypass redundant late-stage refinement. Theoretically, we show that STP reduces rollout sampling cost and characterize the estimation bias–variance trade-off of spatial pruning and the output perturbation induced by temporal pruning. We further relate the bias and variance of paired ELBO-difference estimates to error and variance bounds for GRPO objective. Extensive experiments demonstrate that STP outperforms the evaluated baselines in accuracy and RL training efficiency. Our code is available at https://anonymous.4open.science/r/STP-757E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Large Language Models, Reinforcement Learning, Group Relative Policy Optimization, LLM efficiency, Data-efficient training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QDMAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spatio-Temporal Pruning (STP), a framework intended to make reinforcement learning for masked diffusion language models cheaper and more stable. STP prunes along two axes: spatial pruning fixes a fraction of response tokens to anchor values from earlier generations, and temporal pruning skips the final denoising steps and fills all remaining masks in one parallel forward pass. The paper establishes a lower-variance upper-bound for the ELBO estimator, a bound on the output change from early completion, and bounds linking ELBO-difference error to the GRPO objective.

Critical strengths

The proposed method exploits dLLM structure in a simple and modular way. Both pruning operations use properties unique to masked diffusion models, such as conditioning on arbitrary fixed tokens, and filling all remaining masks in a single forward pass. The authors also provide theoretical analysis on the variance reduction and reduced sampling cost.

Critical weaknesses

The efficiency claims in the experiments are fragile. On GSM8K and Countdown, STP’s training-time advantage over Diffu-GRPO is only 1.4% and 3.4% (Table 1). Those figures also exclude the one-time cost of building the static traces. When it's included, the paper’s own end-to-end numbers are 150,500s on GSM8K and 148,299s on Countdown (line 405), slower than Diffu-GRPO’s 146,641s and 147,100s (Table 1). Also, the accuracy gains on MATH and GSM8K appear marginal, raising questions about whether the reduction in training time achieved by STP justifies the potential loss in accuracy.

Questions for authors
  1. Please use \citep{} and \citet{} correctly.
  2. The title of Section 2.1 should be "GRPO for LLMs", not "GRPO for dLLMs".
  3. On line 182, how is SfixedS_{\text{fixed}} chosen? Is xfixed∈Sfixedx_{\text{fixed}} \in S_{\text{fixed}}? Is γ=γˉ\gamma = \bar{\gamma}?
  4. Diffu-GRPO achieves the same accuracy as the base model on MATH in Table 1, which seems unexpected given the report in the original d1 paper.
  5. The efficiency gains appear marginal, and accuracy may be a more important consideration during training. The longer training time may be an acceptable trade-off if it leads to better accuracy.
  6. The main experiments are conducted on LLaDA-8B-Instruct. Will the authors consider evaluating their method on more advanced DLMs, such as Efficient-DLM?
  7. Surprised to see SPG achieves 0 accuracy in Table 1. (SPG is known to perform well on Countdown. Please double check.
4 comments
AuthorsOpenReview ↗

Response to Reviewer QDMA (Part 1 / 3)

We really appreciate the reviewer for the detailed and constructive feedback. We are glad that you recognized the way our method exploits dLLM structure in a simple and modular manner, and that you found the theoretical analysis on variance reduction and sampling cost useful. We also take seriously your concerns regarding the robustness of our efficiency claims and the plausibility of several reported numbers. We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


Q1. On GSM8K and Countdown, STP’s training-time advantage over Diffu-GRPO is only 1.4% and 3.4% (Table 1). Those figures also exclude the one-time cost of building the static traces. When it's included, the paper’s own end-to-end numbers are 150,500s on GSM8K and 148,299s on Countdown (line 405), slower than Diffu-GRPO’s 146,641s and 147,100s (Table 1).

A1: We sincerely appreciate your valuable comments. We agree that, in a single end-to-end comparison against the fastest baseline, Diffu-GRPO, STP does not have an advantage in end-to-end training time on GSM8K and Countdown once the one-time anchor construction cost is included.

Relative to GRPO with ELBO, after accounting for the anchor construction cost, STP reduces end-to-end training time by 20.0%, 7.7%, and 9.9% on MATH, GSM8K, and Countdown, respectively. Meanwhile, STP achieves accuracies of 36.6%, 81.0%, and 66.0%, respectively, all higher than those of Diffu-GRPO (32.8%, 80.2%, and 25.4%), with the improvement on Countdown being particularly pronounced.

Anchors are task-level static resources shared by all prompts and training runs of the same task, so their construction cost can be amortized across multiple experiments on the same task. In addition, STP is orthogonal to the underlying RL objective and can be stacked on top of Diffu-GRPO. Table 6 verifies this on MATH: with STP added, the online training time of Diffu-GRPO is reduced by 29.8% (excluding anchors) and accuracy improves by 1.6 percentage points; after accounting for the anchor construction cost, the end-to-end training time is still reduced by 25.1% (including anchors).


Q2. Also, the accuracy gains on MATH and GSM8K appear marginal, raising questions about whether the reduction in training time achieved by STP justifies the potential loss in accuracy.

A2: We thank the reviewer for this valuable suggestion. The reduction in training time achieved by STP does not come at the cost of accuracy: STP attains accuracies of 36.6%, 81.0%, and 66.0% on MATH, GSM8K, and Countdown, respectively, improving over GRPO with ELBO by 2.4, 0.5, and 29.7 percentage points. Relative to GRPO with ELBO, after accounting for the anchor construction cost, STP reduces end-to-end training time by 20.0%, 7.7%, and 9.9% on the three tasks, respectively. This efficiency gain arises together with the accuracy gain, consistent with the reduced variance of the ELBO-difference estimator. Figure 2 shows the distribution of the ELBO estimation variance during training, where several high-variance outliers can be observed; high-variance estimates may produce extreme ratios that fall into the clipping region, thereby wasting effective learning signals. By reducing the estimation variance and the frequency of high-variance outliers, STP makes the ratio and the subsequent gradient updates more stable. Table 2 shows that STP reduces the variance of the paired current–old ELBO by 79.1%. These results are consistent with the observation that STP maintains and improves final accuracy while reducing training time.


Q3 and Q4. Please use \citep{} and \citet{} correctly. The title of Section 2.1 should be "GRPO for LLMs", not "GRPO for dLLMs".

A3 and A4: Thanks for the catch. We will correct these typos and conduct a complete check in the revision.


AuthorsOpenReview ↗

Response to Reviewer QDMA (Part 2 / 3)

Q5. On line 182, how is SfixedS_{fixed} chosen? Is xfixed∈Sfixedx_{fixed} \in S_{fixed}? Is γ=γˉ\gamma = \bar{\gamma}?

A5: We consider two ways of choosing SfixedS_{\mathrm{fixed}}: random selection and confidence-based selection. The main experiments adopt Low Confidence Retention, which retains low-confidence tokens as active positions and fixes high-confidence anchor tokens; Random Selection instead randomly selects fixed tokens among candidate positions, with results reported in Table 7.

Sfixed⊆{1,…,L}S_{\mathrm{fixed}}\subseteq\{1,\ldots,L\} denotes the set of fixed token position indices, and xfixed∈VLx_{\mathrm{fixed}}\in\mathcal{V}^{L} denotes the complete anchor sequence. Hence it is not the case that xfixed∈Sfixedx_{\mathrm{fixed}}\in S_{\mathrm{fixed}}; for i∈Sfixedi\in S_{\mathrm{fixed}}, position ii uses the anchor token xfixedix_{\mathrm{fixed}}^{i}, while positions with i∉Sfixedi\notin S_{\mathrm{fixed}} continue to participate in denoising and exploration.

γ\gamma denotes the target spatial pruning ratio, and γˉ=∣Sfixed∣/L\bar{\gamma}=|S_{\mathrm{fixed}}|/L denotes the actual fraction of fixed positions. Under Random Selection, E[γˉ]=γ\mathbb{E}[\bar{\gamma}]=\gamma; under Low Confidence Retention, we select approximately γL\gamma L high-confidence positions to fix, so the actual ratio is affected by the sequence length and rounding. Consequently, γ\gamma and γˉ\bar{\gamma} are typically close but not necessarily exactly equal. In the paper, tstartt_{\mathrm{start}} uses the actual ratio γˉ\bar{\gamma}, i.e., tstart=1−γˉt_{\mathrm{start}}=1-\bar{\gamma}.


Q6. Diffu-GRPO achieves the same accuracy as the base model on MATH in Table 1, which seems unexpected given the report in the original d1 paper.

A6: The fact that Diffu-GRPO matches the base model on MATH is not contradictory to the original paper. The results in the original paper that exceed the base model come from selecting the checkpoint with the highest test accuracy every 100 steps, whereas we report the performance of the final checkpoint, a convention consistent with [1]. Figure 7 of the d1 paper shows that the reward trajectory of LLaDA+Diffu-GRPO during MATH training varies considerably and is below or on par with the initial value for most of the time, indicating that the training process is subject to fluctuations.


Q7. The efficiency gains appear marginal, and accuracy may be a more important consideration during training. The longer training time may be an acceptable trade-off if it leads to better accuracy.

A7: We agree with the reviewer that training efficiency should not come at the expense of accuracy or training stability. The design goal of STP is precisely to simultaneously reduce the computational overhead of rollout and the noise in ELBO estimation, thereby improving optimization while increasing training efficiency.

Theoretically, Theorem 4.1 characterizes the reduction in sampling computation brought by temporal pruning; Theorem 4.2 shows that spatial pruning reduces the variance upper bound of the ELBO estimate to (n/L)2(n/L)^2 times the original; Eqs. (13)–(15) analyze the resulting bias–variance trade-off; and Theorem 4.5 further characterizes the effect of the error and variance of the ELBO difference on the GRPO surrogate. These analyses show that STP can improve training stability by reducing estimation noise, rather than merely reducing computation.

Empirically, STP achieves accuracy no lower than the baselines on all three benchmarks (relative to Diffu-GRPO: +3.8, +0.8, and +40.6 percentage points on MATH, GSM8K, and Countdown, respectively), while reducing the variance of the paired current–old ELBO by 79.1% (Table 2) and requiring shorter training time. Therefore, the efficiency gains of STP are not obtained at the expense of optimization quality: STP achieves more stable training with lower variance, thereby reaching higher accuracy in less time.


AuthorsOpenReview ↗

Response to Reviewer QDMA (Part 3 / 3)

Q8. The main experiments are conducted on LLaDA-8B-Instruct. Will the authors consider evaluating their method on more advanced DLMs, such as Efficient-DLM?

A8: We thank the reviewer for this suggestion. To examine the applicability of STP across different DLM architectures, we further conducted experiments on Efficient-DLM-4B on GSM8K and Countdown. To ensure consistency in comparisons across methods, all settings use the same task prompts, training configurations, and full-sequence sampler. We found that the official Efficient-DLM sampler suffers from severe hallucination on these two datasets, with base accuracy close to 0, making RL training infeasible; we therefore uniformly switched to the full-sequence sampler for decoding.

Table1: Accuracy and online RL training time on Efficient-DLM-4B.

MethodsGSM8K AccuracyGSM8K Time(s)Countdown AccuracyCountdown Time(s)
Efficient-DLM-4B49.6588%-4.2969%-
GRPO w/ ELBO35.9363%543581.5625%63135
SPG1.5163%474820.0000%54275
Diffu-GRPO28.7339%487774.6875%55509
GRPO w/ ELBO+STP(Ours)49.9621%461086.2500%53012

On GSM8K, the base accuracy of Efficient-DLM-4B is 49.66%, which increases to 49.96% with STP; the other three RL methods all exhibit clear degradation relative to the base model. We conjecture that this is because the training signal of this base model on GSM8K is relatively sparse, so methods with larger variance are more affected by noise under limited signal, whereas STP maintains training stability by reducing estimation variance. The online RL training time of STP is 46,108 s; relative to GRPO with ELBO and Diffu-GRPO, STP reduces training time by approximately 15.2% and 5.5%, respectively.

On Countdown, STP improves accuracy from 4.30% for the base model to 6.25%, with a training time of 53,012 s; relative to GRPO with ELBO and Diffu-GRPO, STP reduces training time by approximately 16.0% and 4.5%, respectively.

These results show that STP can likewise improve both training efficiency and final accuracy on Efficient-DLM-4B. This provides additional evidence for the transferability and training stability of STP across different DLM architectures.


Q9. Surprised to see SPG achieves 0 accuracy in Table 1. (SPG is known to perform well on Countdown. Please double check.

A9:We thank the reviewer for this reminder. We have re-checked the implementation, training logs, and evaluation results of SPG. Under the unified experimental setting of this paper, SPG uses the token-level variant, and we report the final checkpoint at the end of training; all three independent runs on Countdown exhibited training collapse, resulting in a final accuracy of 0.0%. SPG removes the KL divergence term and performs μ gradient updates on the same batch of samples while dropping the importance sampling correction, making training more prone to model collapse. This instability is already reflected in the original SPG paper: its Figure 9 shows a sudden increase in generation length during training on Countdown, and issues in the official repository also report training collapse. Since we report the final checkpoint, if collapse occurs in the middle or later stages of training, its state will be directly reflected as 0.0% accuracy on Countdown and below-base performance on GSM8K.

[1] DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. ICLR 2026


We hope the above clarifies the scope of our efficiency claims—STP's advantage is established relative to GRPO with ELBO and, more importantly, it is orthogonal and stackable on top of Diffu-GRPO (Appendix Table 6), where it reduces end-to-end training time by 25.1% with a 1.6-point accuracy gain—and that our re-examination of the SPG and Diffu-GRPO results, together with the additional Efficient-DLM-4B experiments, addresses your concerns about the plausibility and generality of the reported numbers. We would be grateful if you would reconsider the rating should these clarifications and additional results address your concerns. Thank you once again for your careful and constructive review.

Dear Reviewer,

Thank you for your careful review. We have submitted point-by-point responses to your questions and, following your suggestion, added experiments on Efficient-DLM-4B. We would like to invite you to review them at your convenience and let us know whether these clarifications and new results address your main concerns. Following the numbering of your original questions, the key responses are as follows:

  1. On the trade-off between efficiency and accuracy (main concern and Q5). Relative to GRPO w/ ELBO, STP improves accuracy by 2.4, 0.5, and 29.7 percentage points on MATH, GSM8K, and Countdown, respectively; after accounting for anchor costs, total training time is still reduced by 20.0%, 7.7%, and 9.9%, respectively. In these experiments, time savings are accompanied by accuracy improvements. We also explicitly acknowledge that, after accounting for anchors, GRPO w/ ELBO + STP does not outperform Diffu-GRPO in single end-to-end time on GSM8K and Countdown. Furthermore, Table 6 shows that, on MATH, adding STP to Diffu-GRPO still reduces end-to-end time by 25.1% relative to Diffu-GRPO even after accounting for anchor construction costs, while improving accuracy by 1.6 percentage points.

  2. On validation with a more advanced DLM (Q6). We have added experiments on Efficient-DLM-4B. Under the setting where all methods uniformly use the full-sequence sampler, STP achieves 49.96% and 6.25% on GSM8K and Countdown, respectively, exceeding other algorithms, and attains the lowest online training time among the compared RL methods. This directly addresses your suggestion regarding validation on extended models and provides additional evidence for cross-model applicability.

  3. On the results of Diffu-GRPO and SPG (Q4, Q7). We have clarified that this paper adopts the final-checkpoint evaluation protocol and discussed the difference between this and results reported with the best checkpoint. In addition, we re-examined the implementation, training logs, and evaluation of SPG: under the token-level setting of this paper, all three independent runs on Countdown exhibited training collapse, with a final accuracy of 0.0%.

  4. On the choice of fixed positions and notation definitions (Q3). The main experiments fix high-confidence anchor positions while retaining low-confidence positions for continued exploration. We have distinguished the fixed position index set SfixedS_{\mathrm{fixed}} from the full anchor sequence xfixedx_{\mathrm{fixed}}, as well as the target pruning ratio γ\gamma from the actual ratio γˉ=∣Sfixed∣/L\bar{\gamma}=|S_{\mathrm{fixed}}|/L. The target and actual ratios need not be strictly equal; initialization uses the actual ratio, i.e., tstart=1−γˉt_{\mathrm{start}}=1-\bar{\gamma}. Results for the random-selection control are shown in Table 7.

  5. On citation format and section titles (Q1, Q2). We accept your suggestions and have corrected the use of \citep{} and \citet{} in the revision, and corrected the title of Section 2.1 to "GRPO for LLMs".

We hope that these clarifications, especially the Efficient-DLM experiments added at your suggestion and the comparisons that include anchor costs, will help you reassess the empirical contributions of this paper. If key questions remain, please point them out and we will be glad to respond further; if your main concerns have been alleviated, we hope you will reconsider your evaluation and score in light of the above evidence. Thank you again for your time, and we look forward to your feedback.

Reviewer_FJsvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author proposes STP (Spatio-Temporal Pruning), which is a way to make GRPO-style RL for masked diffusion LLMs cheaper and more stable. This approach leads to more stability in training and more efficiency (in terms of real time).

Critical strengths
  1. This paper works on meaningful problems. RL for dLLMs suffers from expensive rollouts and noisy likelihood estimates. STP addresses both with one framework instead of treating them separately.

  2. The proposed approach is easy and effective.

  3. The use of anchor tokens makes sense in order to control noise. Spatial pruning relies on masked models being able to condition on fixed tokens at any position.

Critical weaknesses
  1. The speedups are small. Against the fastest baseline, STP saves 15.9% on MATH but only 1.4% on GSM8K and 3.4% on Countdown. Including the one-time anchor generation, STP is slower than Diffu-GRPO on GSM8K and Countdown. (Efficient RL but not quite efficient in two out of three tasks).

  2. A critical issue regarding SPG. SPG is a token-level re-implementation that gets 0.0% on Countdown and drops below the base model on GSM8K, while SPG’s own paper reports strong Countdown gains. Diffu-GRPO on MATH equals the base model.

Questions for authors
  1. Temporal pruning reduces the number of generation passes by only 4% (128 → 123), but training time drops by 19% (126k → 102k seconds). What accounts for this much larger reduction in training time?

  2. Theorem 4.1 only considers generation steps. How much of the actual training cost comes from Monte Carlo ELBO scoring, and how much does spatial pruning help reduce this overhead?

  3. Anchored rollouts are not sampled directly from �0��0�, but they are still used in the GRPO ratio as if they were on-policy. Why is an off-policy correction unnecessary here? Is there any bias introduced, and if so, how significant is it?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FJsv (Part 1 / 2)

We really appreciate the reviewer for the detailed and constructive feedback. We are pleased that the reviewer found our work addresses a meaningful problem—expensive rollouts and noisy likelihood estimates in RL for dLLMs—and considered STP an easy and effective framework that handles both with one design, with anchor tokens being a sensible way to control noise.

We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


Q1. Against the fastest baseline, STP saves 15.9% on MATH but only 1.4% on GSM8K and 3.4% on Countdown. Including the one-time anchor generation, STP is slower than Diffu-GRPO on GSM8K and Countdown.

A1: We sincerely appreciate these valuable comments. We agree that, in a single end-to-end comparison against the fastest baseline, Diffu-GRPO, STP does not have an advantage in end-to-end training time on GSM8K and Countdown once the one-time anchor construction cost is included.

Relative to GRPO with ELBO, after accounting for the anchor construction cost, STP reduces end-to-end training time by 20.0%, 7.7%, and 9.9% on MATH, GSM8K, and Countdown, respectively. Meanwhile, STP achieves accuracies of 36.6%, 81.0%, and 66.0%, respectively, all higher than those of Diffu-GRPO (32.8%, 80.2%, and 25.4%), with the improvement on Countdown being particularly pronounced.

Anchors are task-level static resources shared by all prompts and training runs of the same task, so their construction cost can be amortized across multiple experiments on the same task. In addition, STP is orthogonal to the underlying RL objective and can be stacked on top of Diffu-GRPO. Table 6 verifies this on MATH: with STP added, the online training time of Diffu-GRPO is reduced by 29.8% (excluding anchors) and accuracy improves by 1.6 percentage points; after accounting for the anchor construction cost, the end-to-end training time is still reduced by 25.1% (including anchors).


Q2. SPG is a token-level re-implementation that gets 0.0% on Countdown and drops below the base model on GSM8K, while SPG’s own paper reports strong Countdown gains. Diffu-GRPO on MATH equals the base model.

A2: This discrepancy mainly stems from differences in the checkpoint selection protocol and implementation settings. The results in the original paper come from selecting the checkpoint with the highest test accuracy every 100 steps, whereas we report the performance of the final checkpoint, a convention consistent with [1].

(i) SPG's 0.0% on Countdown and below-base performance on GSM8K stem from its training instability. SPG removes the KL divergence term and performs μ gradient updates on the same batch of samples while dropping the importance sampling correction, making training more prone to model collapse. This instability is already reflected in the original SPG paper: its Figure 9 shows a sudden increase in generation length during training on Countdown, and issues in the official repository also report training collapse. Since we report the final checkpoint, if collapse occurs in the middle or later stages of training, its state will be directly reflected as 0.0% on Countdown and below-base performance on GSM8K. To maintain consistency with other token-level baselines, we implement the token-level variant of SPG. We note that the seq-level variant is itself more stable than the token-level one [2], so the token-level setting constitutes a more conservative comparison for SPG.

(ii) Diffu-GRPO matching the base model on MATH is not contradictory to the original paper. Figure 7 of the d1 paper shows that, during LLaDA+Diffu-GRPO training on MATH, the reward fluctuates considerably and is below or on par with the initial value for most of the time, which is consistent with our observation that the final checkpoint matches the base model. The results in the original paper that exceed the base model come from selecting the checkpoint with the highest test accuracy every 100 steps. Since the two differ in evaluation protocol, they are not in conflict.


AuthorsOpenReview ↗

Response to Reviewer FJsv (Part 2 / 2)

Q3. Temporal pruning reduces the number of generation passes by only 4% (128 → 123), but training time drops by 19% (126k → 102k seconds). What accounts for this much larger reduction in training time?

A3: Thanks for the question. From the observable system state, the GPU temperature during baseline runs is typically higher than that under the temporal-pruning setting. The baseline needs to perform longer denoising continuously and may thus operate at higher power and temperature states, and in some cases be affected by GPU power or thermal constraints, resulting in lower average SM(Streaming Multiprocessor) clocks; temporal pruning shortens the duration of sustained load, which may allow the retained denoising kernels to maintain higher average SM clocks, thereby amplifying the wall-clock savings. In addition, the actual latency of denoising iterations is not uniform: the 5 skipped iterations may, due to GPU tail effects or because they occur at stages with different GPU clock states, have an average duration above the overall mean, which would also make the wall-clock reduction larger than the NFE reduction. The above analysis is based on existing observations, and fine-grained profiling is left for future work.


Q4. How much of the actual training cost comes from Monte Carlo ELBO scoring, and how much does spatial pruning help reduce this overhead?

A4: We have measured the time proportion of each stage, and the results are shown in the figure below; MC ELBO scoring accounts for approximately 14.3%.

Table1: Proportion of time spent during different stages of the GRPO with ELBO training process on GSM8K.

StageProportion of training time
Rollout generation76.36%
MC ELBO scoring14.27%
Model updates (total)7.17%
Reward computation0.03%
Others2.17%

Spatial pruning does not directly reduce the computational cost of MC ELBO scoring. Its effect lies in the statistical properties of the estimator: Theorem 4.2 shows that the variance upper bound of the ELBO estimate shrinks as (n/L)2(n/L)^2, i.e., the variance is lower for the same number of samples. Indirectly, spatial pruning reduces the overall computational load and may reduce MC ELBO scoring time by improving the GPU power/clock state.


Q5. Anchored rollouts are not sampled directly from πold\pi_{old}, but they are still used in the GRPO ratio as if they were on-policy. Why is an off-policy correction unnecessary here? Is there any bias introduced, and if so, how significant is it?

A5: Thanks for the question. Anchored rollouts are not strictly sampled from πold\pi_{old}; we trade a controllable bias for a reduction in variance, and we provide a theoretical analysis of this trade-off in the paper. We treat anchors as a fixed static prior for each task and keep them unchanged during training; this makes the source of bias fixed within a single training run, so the bias introduced is stationary during training rather than an off-policy bias that drifts with the policy. For a fixed response and anchor set, Theorem 4.2 and Eqs. (13)–(15) analyze that when bS(y)2<vF−vPMb_S(y)^2 < \frac{v_F-v_P}{M}, i.e., when the variance reduction exceeds the squared bias, the MSE of the estimator decreases. Theorem 4.5 further analyzes the effect of the bias and variance of this ELBO difference on the GRPO surrogate: when the variance decreases and the bias is controlled, STP yields tighter error and variance upper bounds for the GRPO surrogate. Empirically, STP reduces the variance of the paired ELBO difference by 79.1% (MATH) while improving accuracy on all three benchmarks, indicating that the resulting trade-off is favorable.

[1] DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation.

[2] Group Sequence Policy Optimization.


We would be grateful if you would reconsider the rating should our responses sufficiently address your soundness concerns, and we welcome any further suggestions on the points that remain open. Thank you once again for your time and constructive feedback.

Dear Reviewer,

Thank you for your specific questions regarding the experimental comparisons and methodological reliability. We have submitted point-by-point responses and would like to further ask you to confirm whether these supplements have alleviated your main concerns. For ease of review, the core responses are summarized below:

  1. On efficiency gains and anchor costs. After accounting for the one-time anchor construction cost, GRPO w/ ELBO + STP in the main table does not outperform Diffu-GRPO in single end-to-end time on GSM8K and Countdown. Relative to its own baseline GRPO w/ ELBO, STP reduces total training time by 20.0%, 7.7%, and 9.9% across the three tasks, respectively, while improving accuracy on all of them. Furthermore, Table 6 verifies the additive benefit of STP on Diffu-GRPO: on MATH, it still saves 25.1% of time after accounting for anchors while improving accuracy by 1.6 percentage points, providing direct evidence of STP's complementary value to existing efficient RL methods.

  2. On the baseline results of SPG and Diffu-GRPO. For Diffu-GRPO, we have clarified that this paper reports the final checkpoint and explained the difference between this protocol and the best-checkpoint results in the original paper. For SPG, we have described the token-level implementation settings and re-examined the implementation, training logs, and evaluation results: under the configuration of this paper, all three independent runs on Countdown exhibited training collapse, with a final accuracy of 0.0%. These clarifications specify the experimental conditions and comparison scope of the relevant results.

  3. On the inconsistency between the number of generations and the reduction in training time (Q1). In our response, we discussed possible contributing factors such as GPU temperature, frequency, and iteration latency differences. Theorem 4.1 characterizes the sampling computation, whereas the wall-clock gains are reported as measured results under the current experimental configuration. The specific source of the discrepancy between the approximately 4% reduction in NFE and the approximately 19% reduction in training time still requires further systematic measurement, which we leave for future work.

  4. On the actual overhead of MC ELBO scoring (Q2). Additional measurements show that, on GSM8K, rollout generation and MC ELBO scoring account for 76.36% and 14.27% of training time, respectively. Spatial pruning does not directly reduce the forward computation of the current scoring implementation; its main role is to improve the estimation properties under the same sampling budget; Theorem 4.2 provides the corresponding variance upper-bound analysis.

  5. On the bias and off-policy issues of anchored rollout (Q3). We acknowledge that anchor conditioning introduces bias, and we adopt biased surrogate optimization in exchange for reduced estimation variance. Section 4 of the paper analyzes this trade-off: Eqs. (13)–(15) present the bias–variance decomposition of the conditional ELBO scoring under fixed responses, as well as the condition under which the variance benefit exceeds the squared bias for MSE improvement; Theorem 4.5 further characterizes the impact of the bias and variance of the ELBO differences on the error and variance upper bounds of the GRPO surrogate objective. Empirically, the paired ELBO difference variance in Table 2 decreases by 79.1%, which, together with the accuracy improvement, supports the empirical benefit of this trade-off.

We would appreciate your feedback at your convenience on whether the above supplements have alleviated your main concerns, and whether there remain any key issues affecting your judgment that require further response from us. We would be happy to prioritize continued discussion around these issues. If these clarifications and empirical results have improved your assessment of the contributions or reliability of this paper, we would also kindly ask you to consider updating your evaluation and score accordingly. Thank you for your time and help.

Reviewer_1FHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spatio-Temporal Pruning (STP) to improve the efficiency of reinforcement learning for diffusion large language models. The method combines separate pruning operations in space and time. The spatial pruning mixes the offline data with online generation, and the temporal pruning stops iterative denoising at a chosen cutoff and complete the remaining masked positions using one parallel prediction pass. They perform theoretical analysis for these two pruning actions. Experimental results demonstrate some improvements in efficiency.

Critical strengths

The idea of combining spatial pruning and temporal pruning is interesting.

Critical weaknesses
  1. The proposed method is mainly a straightforward combination of some existing methods. Novelty in methodology and theoretical analysis is limited. 2. No significant improvement has been demonstrated in both theories and experiments.
Questions for authors

Given in the critical weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1FHk (Part 1 / 2)

We really appreciate the reviewer for the detailed and constructive feedback. We are glad that you found the idea of combining spatial and temporal pruning to be interesting.

We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


Q1. The proposed method is mainly a straightforward combination of some existing methods. Novelty in methodology and theoretical analysis is limited.

A1: We thank the reviewer for this comment. We characterize the methodological contribution of STP as a co-design tailored to the structure of masked dLLM rollout, together with a unified analysis of its efficiency–stability trade-off, rather than as a simple stacking of existing RL pruning methods.

STP represents the rollout of a masked dLLM as a two-dimensional trajectory over "spatial token positions × diffusion time," and designs pruning mechanisms along these two dimensions separately. Spatial pruning exploits the bidirectional inpainting capability of masked dLLMs by using anchor tokens as conditions and generating and conditionally scoring only at positions that require exploration; temporal pruning exploits the parallel masked decoding capability of masked dLLMs by replacing iterative refinement at the tail with a single parallel completion. These two components act on the effective action space and the diffusion trajectory, respectively, and jointly affect the GRPO training signal through conditional ELBO estimation. They rely on the masking, inpainting, and parallel decoding structure of masked dLLMs; transferring them to autoregressive RL would require redefining the corresponding states, actions, and likelihood estimation.

Compared with existing methods that improve rollout or estimators, the distinction of STP lies in handling both spatial positions and diffusion time simultaneously, and in analyzing the joint effect of these two types of pruning on RL training. Theoretically, Theorem 4.1 establishes the reduction in sampling complexity, Theorem 4.2 and Eqs. (13)–(15) characterize the bias–variance trade-off of spatial pruning, Theorem 4.3 provides the output perturbation bound introduced by temporal pruning, and Theorem 4.5 further relates the error and variance of the ELBO difference to upper bounds on the error and variance of the GRPO surrogate.

Since STP operates at the level of rollout and conditional scoring trajectories, it is relatively decoupled from the specific policy-gradient objective. The results in Table 6 on MATH show that STP can be stacked on top of methods such as Diffu-GRPO, SPG, and SDMC, reducing training time while maintaining or improving accuracy.


AuthorsOpenReview ↗

Response to Reviewer 1FHk (Part 2 / 2)

Q2. No significant improvement has been demonstrated in both theories and experiments.

A2: The core contribution of STP is to simultaneously improve the efficiency and optimization stability of dLLM RL training, and to establish a theoretical connection between the pruning mechanisms and the stability of the RL surrogate.

Theoretically, Theorem 4.1 establishes a reduction in rollout sampling cost; Theorem 4.2 and Eqs. (13)–(15) characterize the bias–variance/MSE trade-off of spatial pruning; Theorem 4.3 provides the output perturbation bound of temporal pruning; and Theorem 4.5 propagates the bias and variance of the paired ELBO difference to error and variance bounds on the GRPO surrogate. These results establish a connection among pruning, ELBO estimation, and the stability of GRPO optimization.

Experimental results show that, relative to GRPO with ELBO, STP simultaneously improves accuracy and reduces training time. In Table 1, STP improves accuracy by 2.4, 0.5, and 29.7 percentage points on MATH, GSM8K, and Countdown, respectively, while reducing online RL training time by 24.3%, 11.3%, and 13.7%, respectively. After accounting for the one-time anchor construction cost, the end-to-end training time is still reduced by 20.0%, 7.7%, and 9.9%, respectively. Table 2 further shows that, over the first 2,000 training steps on MATH, STP reduces the variance of the paired current–old ELBO difference from 1.97×10−41.97\times10^{-4} to 4.11×10−54.11\times10^{-5}, a reduction of 79.1%; meanwhile, the average NFE decreases from 128.0 to 108.3, and accuracy increases from 34.0% to 36.0%. Figure 2 illustrates the evolution of the variance during training and its distribution, where STP exhibits fewer high-variance outliers in the variance distribution. Since the GRPO importance ratio is obtained by an exponential transformation of the ELBO difference, high-variance estimates may push the ratio into the clipping region, thereby wasting effective learning signals; STP's suppression of estimation variance helps obtain more stable ratios and gradient updates.

We further conducted additional experiments on Efficient-DLM-4B. Under a unified full-sequence sampler, STP achieves 49.96% accuracy and 46,108 s of online RL training time on GSM8K, an improvement of 0.30 percentage points over the base model's 49.66%; on Countdown, it achieves 6.25% accuracy and 53,012 s, an improvement of 1.95 percentage points over the base model's 4.30%, and attains the lowest online training time on both tasks. These results provide additional evidence for the efficiency and robustness of STP across different DLM architectures.

Table1: Accuracy and online RL training time on Efficient-DLM-4B.

MethodsGSM8K AccuracyGSM8K Time(s)Countdown AccuracyCountdown Time(s)
Efficient-DLM-4B49.6588%4.2969%
GRPO w/ ELBO35.9363%543581.5625%63135
SPG1.5163%474820.0000%54275
Diffu-GRPO28.7339%487774.6875%55509
GRPO w/ ELBO+STP(Ours)49.9621%461086.2500%53012

We hope our responses have clarified that STP is a jointly designed co-design rather than a simple stacking of existing pruning methods, and that its theoretical characterization and empirical gains together constitute a substantive contribution. We understand that novelty and significance were your main concerns, and we would be grateful if you would consider re-evaluating the rating in light of the clarifications and additional results above. Thank you once again for your time and constructive feedback—we look forward to incorporating these discussions into the final version.

Dear Reviewer,

Thank you for your prior review and suggestions. We would like to briefly follow up on the rebuttal we have submitted and kindly ask you to review it at your convenience and let us know whether these clarifications address your main concerns regarding novelty and the magnitude of the contribution. For quick verification, we summarize the key responses below:

  1. On Q1: Novelty of the method and theory. We further clarify the positioning of STP's contribution: for the "token position × diffusion time" trajectory of masked dLLMs, we jointly design anchor-conditioned spatial pruning and temporal pruning via terminal parallel completion, and analyze their impact on the conditional ELBO estimation and the GRPO surrogate objective. Theorems 4.1–4.3 and 4.5, together with Eqs. (13)–(15), respectively characterize the sampling cost, the bias–variance trade-off, the output perturbation, and the propagation of estimation error to the GRPO surrogate objective, constituting the theoretical analysis for this design.

  2. On Q2: Whether the improvements have practical value. Relative to GRPO w/ ELBO, STP improves accuracy by 2.4, 0.5, and 29.7 percentage points on MATH, GSM8K, and Countdown, respectively; after accounting for the one-time anchor construction cost, total training time is still reduced by 20.0%, 7.7%, and 9.9%, respectively (Table 1 and adjacent text). In addition, Table 2 shows that, over the first 2,000 steps on MATH, the variance of the paired current–old ELBO difference decreases from 1.97×10−41.97\times10^{-4} to 4.11×10−54.11\times10^{-5}, a reduction of 79.1%, providing empirical evidence for improved estimation stability.

  3. On the complementary value relative to existing methods. Table 6 verifies the compatibility of STP with Diffu-GRPO, SPG, and SDMC. In particular, on MATH, adding STP improves the accuracy of Diffu-GRPO by 1.6 percentage points, and total training time after accounting for anchor costs is still reduced by 25.1%, indicating that its benefits can be stacked on top of existing methods.

We hope that the above clarifications and results will help you reassess the novelty and practical contribution of this paper. Please let us know whether any key issues affecting your judgment remain, and we would be glad to provide further targeted responses; if your main concerns have been alleviated, we would also kindly ask you to reconsider your evaluation and score in light of this evidence. Thank you again for your time, and we look forward to your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Friendly Reminder

Dear Reviewers,

Thank you for your thoughtful feedback and suggestions, which have helped us further strengthen our study. We have provided detailed responses and conducted additional experiments to address the concerns raised. We would appreciate your feedback on whether our responses and additional results have adequately addressed your concerns.

If any questions remain, we would be glad to provide further clarification. We would also be grateful if you could consider the additional evidence when finalizing your assessment. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.