acceptodds
Under review as a conference paper at ICLR 2027

Online Timestep Allocation with Control Variates for Masked Diffusion Language Models

Abstract

Masked diffusion language models exhibit substantial variation across corruption timesteps, mask patterns, and data samples, making uniform timestep sampling potentially inefficient. We propose Importance Sampling with Control Variates (ISCV), an online, single-forward controller that adapts a full-support timestep proposal using confidence statistics already available in the standard model pass. We show that, with a control variate, the variance-optimal proposal is determined by the second moment of the residualized loss rather than the raw loss. This motivates ISCV-SG, which stop-gradients the correction so that the residual statistics guide future timestep allocation while preserving the expected gradient of the original objective under exact importance weighting. Across five LoRA adaptation tasks, ISCV-SG improves the three-seed mean over uniform training and remains competitive with strong variance-reduction baselines, while retaining runtime close to Standard and requiring approximately half the training time of the strongest baseline P-POTS+MIRROR. Matched ablations and controlled full-parameter pretraining further support the proposed allocation mechanism beyond downstream LoRA adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
masked diffusion language models, importance sampling, control variates, adaptive timestep sampling, variance reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BG8kOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ISCV, an adaptive timestep sampling method for masked diffusion language models. The main motivation is that loss statistics vary considerably across timesteps, making uniform sampling potentially inefficient. The method combines importance sampling with an entropy-based control variate and updates the sampling distribution online without additional forward passes. The authors also consider a stop-gradient variant (ISCV-SG) to preserve the original expected gradient. Experiments on five tasks using LoRA adaptation of LLaDA-8B show consistent improvements over uniform sampling, with competitive performance against stronger baselines at a lower training cost.

Critical strengths
  • The method is practical and relatively efficient. I like that the approach focuses on improving timestep sampling without adding much computational overhead. Its runtime is close to uniform sampling and substantially lower than P-POTS+MIRROR.
  • The improvements over uniform sampling are consistent. The method shows gains across all five tasks, which is encouraging and suggests that the approach can be useful in different settings.
  • The experimental reporting is fairly transparent. Results are provided for individual seeds, and the authors acknowledge that the comparison with the strongest baseline varies across tasks.
Critical weaknesses
  • The improvements over strong baselines seem relatively modest. The gains appear small compared to the variation across random seeds, making it difficult to assess how consistent they are. Reporting standard deviations would help clarify this. I am also somewhat concerned about the number of hyperparameters involved, particularly since some are tuned for individual tasks. It would be useful to know how well the method works with a shared configuration or across different backbones.

  • The advantage of online adaptation is not entirely clear. Since importance weighting preserves the original objective, an important question is how much the adaptive sampling strategy improves over a well-chosen fixed schedule. I would have liked to see more direct comparisons with existing timestep sampling methods, or even a version of ISCV with its learned proposal frozen. This would help establish whether online adaptation itself is responsible for the improvements.

  • It is unclear how much the adjusted-loss sampling contributes to the final results. One thing I found somewhat surprising is that CV seems to provide much of the improvement in Table 2, while ISCV-SG and IS achieve nearly identical results in Table 1. This makes it difficult to tell how much benefit comes specifically from using adjusted rather than raw loss statistics for sampling. A more direct comparison across tasks would help clarify the contribution of this idea.

Questions for authors

N/A

3 comments

W1: Our main advantage is higher accuracy at essentially the same training time as Standard, and competitive accuracy at approximately half the training time of P-POTS+MIRROR. To address variation across seeds, we summarize the original manuscript results from Tables 1 and 9 as three-seed mean ± sample standard deviation. For each task, we select the highest-mean external baseline among Clipped, P-POTS, and P-POTS+MIRROR.

DatasetStrongest external baselineBaseline (mean ± SD)ISCV-SG (mean ± SD)
GSM8KP-POTS+MIRROR65.45 ± 1.2065.66 ± 1.75
MATHClipped36.09 ± 0.8237.03 ± 1.73
HiTabP-POTS+MIRROR61.40 ± 0.4661.80 ± 0.28
OpenScienceP-POTS+MIRROR67.80 ± 0.4667.00 ± 0.36
MBPPP-POTS41.89 ± 0.5942.15 ± 1.57

At comparable training time, ISCV-SG improves over Standard on all 12 same-seed comparisons across the four tasks with reported runtimes: GSM8K, MATH, HiTab, and OpenScience. The mean accuracy gains are +4.02, +2.65, +5.24, and +3.18 pp, respectively, with only 0.5–1.4% additional training time. These results show consistent accuracy gains with negligible training overhead.

At the matched token/update budgets used in the paper, ISCV-SG uses approximately 45–50% of P-POTS+MIRROR's measured training time on these four tasks, while achieving a higher mean accuracy on three of them. Across all five tasks, including MBPP, ISCV-SG has a higher mean than P-POTS+MIRROR on four of five tasks. Thus, P-POTS+MIRROR requires roughly twice the measured training time for comparable accuracy, demonstrating ISCV-SG's advantage in the accuracy–runtime trade-off.

Regarding hyperparameters, within each task, the compared methods use the same LoRA configuration and learning-rate schedule. For the ISCV controller, timestep binning and the default settings for estimating CV coefficients are shared across the reported LoRA experiments. The task-specific choices concern the correction-strength cap, adaptive sampling mass, and IS activation timing. Different dataset sizes and training horizons mean that the same absolute activation step can correspond to different training progress.

Appendix B.12 provides the following local seed-42 sensitivity measurements:

DatasetController settingSeed 42 accuracy (%)
GSM8KCorrection-strength cap λmax⁡=0.50\lambda_{\max}=0.5066.11
Correction-strength cap λmax⁡=0.55\lambda_{\max}=0.5566.64
Correction-strength cap λmax⁡=0.60\lambda_{\max}=0.6064.90
GSM8KAdaptive mass = 5%63.84
Adaptive mass = 10%64.22
Adaptive mass = 20%64.59
OpenScienceIS activation step = 48067.10
IS activation step = 60067.10

Increasing adaptive probability mass fourfold, from 5% to 20%, changes accuracy by 0.75 pp. Delaying IS activation from step 480 to 600 leaves the OpenScience score unchanged at 67.10%. The correction-cap sweep spans 1.74 pp. These measurements support local robustness over the tested settings, particularly for adaptive sampling mass and activation timing. The proposal-probability floor and importance-weight clipping did not activate in the all runs.

For different backbones, we evaluate LLaDA-8B with LoRA adaptation and 169M DiT with full-parameter pretraining. On the latter, the two-seed mean perplexity improves from 74.97 to 73.18 at 1B tokens, supporting applicability beyond LoRA adaptation on a single backbone.

W2: The submission compares against P-POTS, which fits a timestep proposal before training and keeps it fixed (Jia et al., Section 3.1.1). In Tables 1 and 9, ISCV-SG's mean differences from P-POTS are +0.76 pp on GSM8K, +2.77 on HiTab, −0.03 on OpenScience, and +0.26 on MBPP. These compare complete methods; the following within-ISCV ablation directly examines continued adaptation.

We completed three-seed Online/Frozen comparisons on GSM8K and MBPP. Online uses the original manuscript runs; Frozen reports additional runs under matched training and evaluation protocols.

Freeze protocol. On both tasks, after completing epoch 3, Frozen fixes the full sampling distribution, including its uniform-mixture component, from epoch 4 onward. Model and optimizer updates continue, and importance weights use the fixed distribution: w(k)=p(k)/qfreeze(k)w(k)=p(k)/q_{\mathrm{freeze}}(k). Online continues updating its sampling distribution. Training budgets, evaluation protocols, and checkpoint-selection rules are matched.

Online versus Frozen proposal: task performance (%).

DatasetMethodSeed 42Seed 731Seed 20231Avg.
GSM8KISCV-SG (Online)67.6365.0564.2965.66 ± 1.75
ISCV-SG (Frozen)65.5064.6763.6864.62 ± 0.91
MBPPISCV-SG (Online)40.4742.4143.5842.15 ± 1.57
ISCV-SG (Frozen)40.4741.2542.4141.37 ± 0.98

Avg. reports mean ± sample SD across three seeds, calculated from unrounded accuracies. Online improves on all three GSM8K seeds, with a mean gain of 1.04 pp. On MBPP, Online ties Frozen for seed 42 and improves for seeds 731 and 20231, with a mean gain of 0.78 pp.

W3: Our controlled HiTab experiment directly isolates the contribution of using adjusted rather than raw loss statistics: SG-Adjusted improves over SG-Raw by 2.23 percentage points in three-seed mean accuracy, with gains on all three seeds.

Both variants use the same SG implementation and retain the same detached CV correction in the training scalar. Within each seed, initialization, data order, optimizer, training budget, uniform mixture, proposal-update schedule, and evaluation are fixed. The only change is the statistic used to update the timestep sampling distribution: raw-loss statistics for SG-Raw and confidence-corrected loss statistics for SG-Adjusted.

Controlled HiTab comparison: accuracy (%).

MethodSeed 42Seed 731Seed 20231Mean
SG-Raw59.3459.7859.5959.57
SG-Adjusted61.5062.0561.8661.80
Gain (percentage points)+2.16+2.27+2.27+2.23

This also distinguishes the sampling contribution from the CV benefit in Table 2. The CV result in Table 2 belongs to the differentiable S setting, where the correction contributes to the current gradient. In the controlled HiTab SG comparison, the complete correction is detached in both variants: for the same sampled loss and importance weight, both produce the gradient w∇θℓw\nabla_\theta\ell. The observed improvement therefore comes from changing the statistics that guide subsequent timestep allocation, rather than adding a correction-gradient term.

The original manuscript results provide additional evidence on the other tasks:

Original IS versus ISCV-SG results: three-seed mean accuracy (%), Tables 1, 2, and 9.

DatasetISISCV-SGDifference (percentage points)
GSM8K65.6865.66−0.02
OpenScience66.6067.00+0.40
MBPP41.2542.15+0.91

Differences are calculated before rounding the displayed means. The near tie occurs on GSM8K, while OpenScience and MBPP show positive mean gains. These are the original manuscript comparisons; the additional statistic-only SG experiment was conducted on HiTab.

The new HiTab SG-Raw runs are independent of the historical HiTab IS runs under the same experimental protocol. Historical IS uses raw-loss statistics and has no CV term in its training scalar; both variants in the new SG comparison retain the detached CV term. The +2.23 pp gain across all three HiTab seeds thus provides direct training evidence for the value of confidence-corrected sampling statistics within SG.

Thank you to the authors for the detailed rebuttal and additional experiments.

I appreciate the new controlled experiment addressing W3, which provides convincing evidence for the contribution of confidence-corrected sampling statistics.

However, my main concerns regarding W1 and W2 remain. The performance improvements over strong baselines and the gains from online adaptation are still relatively marginal, particularly in light of the reported standard deviations. While I acknowledge the computational efficiency of the proposed method, I remain unconvinced that these modest improvements sufficiently establish its significance and novelty over existing timestep sampling approaches.

Therefore, I will maintain my original rating.

Reviewer_1gnJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes a novel computing-allocation method for masked diffusion language models using a control-variate method. The method, termed Importance Sampling with Control Variates (ISCV), performs masking probability estimation using the control variate for efficient importance sampling. The manuscript first discusses the ideal estimation of confidence statistic for mask allocation, and then introduces an online approximation algorithm for training. Experiments in five benchmarks, i.e., GSM8K, MATH, OpenScienceReasoning-2, HiTab, and MBPP in response-only LoRA setting show that ISCV outperformed standard-training DLM with margins and work comparably with variance-reduction P-POTS+MIRROR that nearly double the runtime.

Critical strengths
  • The method is technically sound by enabling variate control without increasing inference complexity.
  • Promising empirical results: ISCV outperformed standard-training DLM with margins and is competitive with slow P-POTS+MIRROR.
Critical weaknesses
  • Background knowledge: I needed to do some background research to grasp the design and purpose of "the classical control-variate estimator" in the head of Sec. 3. Having a brief background section would be beneficial.

  • Comparisons with recent importance-sampling-related DLMs: Importance sampling in DLM is incorporated for several training methods [a, b]. I would see some conceptual differences, but they should be discussed and ideally compared experimentally. Currently, the combination of IS and CV looks like incremental engineering improvements rather than a key technical innovation.

    a: Diffusion Language Model Parallel Decoding via Product-of-Experts Bridge, ICML2026 b: GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models, ICLR workshop 2026

  • Limited improvement over vanilla CV: Table 2 properly ablates IS and CV, but the CV and the proposed ISCV-S work are considerably similar. The necessity of IS would be discussed more deeply.

Questions for authors
  • Full training results look limited in perplexity-based evaluation rather than task performance, seemingly due to limited training budgets. If any progress in these results is achieved, I am welcome for updates. At least, it is recommended to include the expected advantage over LoRA-based incorporation of ISCV.
4 comments

W1: We will add the following short introduction to Section 3:

A control variate reduces noise using a correlated auxiliary statistic. If a loss ℓ\ell and an inexpensive statistic hh fluctuate together, the estimator ℓ−β(h−E[h])\ell-\beta(h-\mathbb E[h]) preserves the expected loss while removing some of its predictable fluctuation. With a suitable coefficient β\beta, this reduces variance. ISCV uses predictive confidence as the auxiliary statistic, obtained from the existing model forward pass.

W2: PoE-Bridge operates during decoding: a DLM proposes continuations, and a product-of-experts bridge helps correct them toward an autoregressive target. ISCV operates during training, allocating samples across diffusion timesteps without an autoregressive target or a change to decoding.

GIFT operates during fine-tuning. It uses a fully masked response pass to estimate predictive entropy and set token-dependent corruption rates, followed by a training-loss pass. ISCV retains the conditional mask-sampling procedure and updates a global timestep proposal using statistics from the existing training pass. Thus, GIFT adapts token-level corruption, whereas ISCV adapts timestep allocation. We could not identify a released official implementation of GIFT and have not completed a matched-budget comparison.

Our technical contribution is an adjusted-estimator-aware timestep proposal, updated online from a single standard training forward pass. For fixed model and correction coefficients, the ideal scalar-estimator allocation is q(k)∝E[ℓ~2∣k]q(k)\propto\sqrt{\mathbb E[\widetilde\ell^2\mid k]}. Allocation therefore accounts for the loss statistics after confidence correction, coupling IS and CV through the proposal. The controlled HiTab SG ablation supports this design: changing only the proposal statistics from raw to adjusted losses improves three-seed mean accuracy by 2.23 pp, with gains on all three seeds.

W3: Adding IS improves the reported three-seed mean on all four tasks. Tables 2 and 9 provide the following comparisons:

CV and ISCV-S task performance (%), Tables 2 and 9.

DatasetMethodSeed 42Seed 731Seed 20231Avg.
GSM8KCV68.0167.1766.3467.17
ISCV-S68.3967.2467.3267.65
HiTabCV58.9662.0762.1261.05
ISCV-S62.0263.3963.2662.89
OpenScienceCV66.9066.7066.3066.63
ISCV-S69.6067.2068.5068.43
MBPPCV41.6340.8640.4740.99
ISCV-S43.1942.4142.8042.80

The mean gains are +0.48, +1.84, +1.80, and +1.81 pp, respectively; ISCV-S exceeds CV in all 12 reported same-seed comparisons. These results support their complementary value in experiments.

CV reduces fluctuations within a timestep, while IS allocates samples across timesteps. In the ideal centered scalar-estimator setting, writing μk=E[ℓ∣k]\mu_k=\mathbb E[\ell\mid k] gives E[ℓ~2∣k]=μk2+Var⁡(ℓ~∣k)\mathbb E[\widetilde\ell^2\mid k]=\mu_k^2+\operatorname{Var}(\widetilde\ell\mid k). These second moments can still differ across timesteps after CV adjustment. Updating the proposal from the adjusted statistics therefore addresses sampling inefficiency that within-timestep correction alone does not resolve.

Q1: We have added downstream evaluations of the 169M-parameter MDLM after 1B tokens of full-parameter training, using checkpoints from seeds 42, 731, and 20231. ISCV-SG improves mean LAMBADA accuracy at all four evaluated CFG settings, with gains of 0.35–0.60 percentage points.

Compared with LoRA, full-parameter training allows ISCV's timestep allocation to affect updates to all model weights, extending its expected optimization benefit beyond the low-rank adapter parameters. In our 169M full-parameter experiments, the previously reported two-seed mean generative perplexity, using seeds 42 and 731, improves from 74.97 to 73.18 at 1B tokens under matched token and update budgets.

We evaluate all 5,153 LAMBADA examples with confidence-ordered greedy masked-cloze decoding, BOS, and MASK padding to length 512. The reference answer's token count sets the answer-slot length, but its tokens are not supplied. Accuracy requires an exact match of the target token sequence.

LAMBADA exact-match accuracy (%) at 1B tokens.

DatasetMethodSeed 42Seed 731Seed 20231Avg.
LAMBADA (CFG=0)Standard6.715.419.517.21 ± 2.09
ISCV-SG9.003.929.927.61 ± 3.23
LAMBADA (CFG=0.3)Standard9.288.1313.1010.17 ± 2.60
ISCV-SG12.215.7813.5610.52 ± 4.16
LAMBADA (CFG=0.6)Standard11.009.8814.2411.71 ± 2.27
ISCV-SG13.997.0415.2012.08 ± 4.40
LAMBADA (CFG=1)Standard11.6210.7715.2512.55 ± 2.38
ISCV-SG15.147.7216.5913.15 ± 4.76

Avg. reports mean ± sample SD across three seeds. The mean gains are +0.40, +0.35, +0.37, and +0.60 pp for CFG=0/0.3/0.6/1.0, respectively, calculated before rounding. Seeds 42 and 20231 improve at every CFG value; seed 731 favors Standard. Given the limited training budget of 1B tokens, incomplete convergence may contribute to the observed seed-to-seed variation.

Dear Reviewer,

The discussion deadline is approaching. Please review the authors’ responses and clearly indicate whether they adequately address your concerns. Kindly update your score if appropriate.

Best regards,

AC

I appreciate the authors response, and apologize for my late reply. Among my concerns, W1 is solved and and W2 is half solved: While the differences between existing IS-based DLM were clarified, I am not fully convinced by the significance of the technical contribution, i.e., adjusted-estimator-aware timestep proposal, especially after given multiple IS-based existing DLM methods. Empirical gains of performance are modest, so I think the results are not conclusive for acceptance by themselves. For W3, the improvements look still marginal in relative to the seed-based variations; but I do not feel too much negative about this because large-budget retraining would be impossible in the rebuttal period. I will reread the paper and other reviews and finish my recommendation during the reviewer-AC discussion.

Reviewer_sYuKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ISCV uses confidence stats from the model's existing forward pass to decide how often to sample each timestep during masked diffusion language-model training. It updates the sampling distribution from second moments of a loss adjusted by a control variate. For this, the authors propose 2 variants: ISCV-S differentiates through this correction while ISCV-SG detaches it using a stop-grad and uses importance weights to preserve the original expected gradient. The core aim is to improve training allocation at little extra cost. The experiments cover three-seed adaptation on five tasks, matched S/SG comparisons, component ablations, gradient probes, and two-seed full-parameter pretraining.

Critical strengths
  1. The approach provides a useful tradeoff of efficiency and computational requirements. The ISCV-SG approach provides an improvement of 1.68-5.24 points in the three-seed mean compared to uniform training on all five tasks. In terms of execution time, ISCV-SG is close to Standard within 1.4% on the four timed tasks (Table 1, page 8). It is comparable to the more computationally costly paired mask baseline with similar token and update budgets. This is a useful application of known techniques: confidence drives sampling in the future without additional model evaluations and changing the objective of SG. Full-parameter pretraining (Tables 6/8) also provides some evidence other than LoRA.

  2. Multiple experiments validate the proposed method. The matched S/SG test (Table 3) isolates the impact of differentiation through the correction from its use merely as guidance in sampling. S outperforms Standard in all GSM8K and HiTab seeds, while SG still outperforms Standard in all seeds on average. Table 13 validates the choice of adjusted over unadjusted proposal statistics in S. For SG, the probes of all trainable LoRA parameters show preference for the corrected proposal; the median ratio of predicted gradient-second moment to uniform is 0.586 for the corrected proposal versus 0.837 for the unadjusted proposal (Tables 5/11).

Critical weaknesses
  1. It is not clear how the additional loss term affects ordinary online imp. sampling in SG. As it follows from Eqn. (10), the derivative grad(s)=w grad(ell). The correction can influence future sampling in such a way as to make sampling yield beneficial raw gradients, because it is independent from the sampled trajectory. Checkpoint probes suggest this hypothesis, however, the training ablation in Table 13 applies S, where the confidence terms change derivatives. Still, there are encouraging results for SG versus IS on HiTab (+2.99 points), OpenScience (+0.40), and MBPP (+0.91), with the almost equal result for GSM8K (see Tables 1/2/9). It's not clear if all other settings of the controller are the same. In HiTab, contrast the performance on two separate SG runs which have only one different setting – if the time-step probability estimation is done by raw or by confidence-corrected losses. Make sure everything else stays the same (initialization, data ordering, optimizer, training time, mixture for uniform sampling, proposal update schedule). This would show whether confidence adds value beyond the same sampling procedure without the adjustment.
Questions for authors

Point 1 from Critical Weaknesses

2 comments

W1: We completed the requested three-seed HiTab comparison: SG-Adjusted improves over SG-Raw by 2.23 percentage points on average, with gains on all three seeds.

Both variants use the same SG implementation and retain the CV correction in the importance-weighted training scalar, with the complete correction detached during backpropagation. SG-Raw updates the sampling distribution using raw-loss statistics; SG-Adjusted uses cv-corrected loss statistics. Within each seed, initialization, data order, optimizer, training budget, uniform mixture, proposal-update schedule, and evaluation are fixed; only the proposal-statistic source changes.

Controlled HiTab SG-Raw / SG-Adjusted performance (%).

DatasetMethodSeed 42Seed 731Seed 20231Avg.
HiTabSG-Raw59.3459.7859.5959.57
SG-Adjusted61.5062.0561.8661.80

The distinction between the historical IS baseline and the two SG variants is:

MethodStatistics used to update the proposalBackpropagated training scalar
Historical ISRaw-loss statisticswℓw\ell
SG-RawRaw-loss statisticswℓ−stopgrad⁡(wβc)w\ell-\operatorname{stopgrad}(w\beta c)
SG-AdjustedCV-adjusted loss statisticswℓ−stopgrad⁡(wβc)w\ell-\operatorname{stopgrad}(w\beta c)

Here cc denotes the confidence statistic, and the importance weight ww is detached. For the same sampled loss and weight, both scalar forms give the parameter gradient w∇θℓw\nabla_\theta\ell. The controlled SG comparison therefore tests the effect of the adjusted statistics on subsequent timestep allocation.

The additional SG-Raw runs are under the same experimental protocol with the historical IS runs in manuscript Table 9.

Thanks for the clear response and the additional experiments. These directly address my concern on isolating the role of the cv-corrected loss. I will update my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.