- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ISCV, an adaptive timestep sampling method for masked diffusion language models. The main motivation is that loss statistics vary considerably across timesteps, making uniform sampling potentially inefficient. The method combines importance sampling with an entropy-based control variate and updates the sampling distribution online without additional forward passes. The authors also consider a stop-gradient variant (ISCV-SG) to preserve the original expected gradient. Experiments on five tasks using LoRA adaptation of LLaDA-8B show consistent improvements over uniform sampling, with competitive performance against stronger baselines at a lower training cost.
Critical strengths
- The method is practical and relatively efficient. I like that the approach focuses on improving timestep sampling without adding much computational overhead. Its runtime is close to uniform sampling and substantially lower than P-POTS+MIRROR.
- The improvements over uniform sampling are consistent. The method shows gains across all five tasks, which is encouraging and suggests that the approach can be useful in different settings.
- The experimental reporting is fairly transparent. Results are provided for individual seeds, and the authors acknowledge that the comparison with the strongest baseline varies across tasks.
Critical weaknesses
-
The improvements over strong baselines seem relatively modest. The gains appear small compared to the variation across random seeds, making it difficult to assess how consistent they are. Reporting standard deviations would help clarify this. I am also somewhat concerned about the number of hyperparameters involved, particularly since some are tuned for individual tasks. It would be useful to know how well the method works with a shared configuration or across different backbones.
-
The advantage of online adaptation is not entirely clear. Since importance weighting preserves the original objective, an important question is how much the adaptive sampling strategy improves over a well-chosen fixed schedule. I would have liked to see more direct comparisons with existing timestep sampling methods, or even a version of ISCV with its learned proposal frozen. This would help establish whether online adaptation itself is responsible for the improvements.
-
It is unclear how much the adjusted-loss sampling contributes to the final results. One thing I found somewhat surprising is that CV seems to provide much of the improvement in Table 2, while ISCV-SG and IS achieve nearly identical results in Table 1. This makes it difficult to tell how much benefit comes specifically from using adjusted rather than raw loss statistics for sampling. A more direct comparison across tasks would help clarify the contribution of this idea.
Questions for authors
N/A
3 comments
W2: The submission compares against P-POTS, which fits a timestep proposal before training and keeps it fixed (Jia et al., Section 3.1.1). In Tables 1 and 9, ISCV-SG's mean differences from P-POTS are +0.76 pp on GSM8K, +2.77 on HiTab, −0.03 on OpenScience, and +0.26 on MBPP. These compare complete methods; the following within-ISCV ablation directly examines continued adaptation.
We completed three-seed Online/Frozen comparisons on GSM8K and MBPP. Online uses the original manuscript runs; Frozen reports additional runs under matched training and evaluation protocols.
Freeze protocol. On both tasks, after completing epoch 3, Frozen fixes the full sampling distribution, including its uniform-mixture component, from epoch 4 onward. Model and optimizer updates continue, and importance weights use the fixed distribution: . Online continues updating its sampling distribution. Training budgets, evaluation protocols, and checkpoint-selection rules are matched.
Online versus Frozen proposal: task performance (%).
| Dataset | Method | Seed 42 | Seed 731 | Seed 20231 | Avg. |
|---|---|---|---|---|---|
| GSM8K | ISCV-SG (Online) | 67.63 | 65.05 | 64.29 | 65.66 ± 1.75 |
| ISCV-SG (Frozen) | 65.50 | 64.67 | 63.68 | 64.62 ± 0.91 | |
| MBPP | ISCV-SG (Online) | 40.47 | 42.41 | 43.58 | 42.15 ± 1.57 |
| ISCV-SG (Frozen) | 40.47 | 41.25 | 42.41 | 41.37 ± 0.98 |
Avg. reports mean ± sample SD across three seeds, calculated from unrounded accuracies. Online improves on all three GSM8K seeds, with a mean gain of 1.04 pp. On MBPP, Online ties Frozen for seed 42 and improves for seeds 731 and 20231, with a mean gain of 0.78 pp.
W3: Our controlled HiTab experiment directly isolates the contribution of using adjusted rather than raw loss statistics: SG-Adjusted improves over SG-Raw by 2.23 percentage points in three-seed mean accuracy, with gains on all three seeds.
Both variants use the same SG implementation and retain the same detached CV correction in the training scalar. Within each seed, initialization, data order, optimizer, training budget, uniform mixture, proposal-update schedule, and evaluation are fixed. The only change is the statistic used to update the timestep sampling distribution: raw-loss statistics for SG-Raw and confidence-corrected loss statistics for SG-Adjusted.
Controlled HiTab comparison: accuracy (%).
| Method | Seed 42 | Seed 731 | Seed 20231 | Mean |
|---|---|---|---|---|
| SG-Raw | 59.34 | 59.78 | 59.59 | 59.57 |
| SG-Adjusted | 61.50 | 62.05 | 61.86 | 61.80 |
| Gain (percentage points) | +2.16 | +2.27 | +2.27 | +2.23 |
This also distinguishes the sampling contribution from the CV benefit in Table 2. The CV result in Table 2 belongs to the differentiable S setting, where the correction contributes to the current gradient. In the controlled HiTab SG comparison, the complete correction is detached in both variants: for the same sampled loss and importance weight, both produce the gradient . The observed improvement therefore comes from changing the statistics that guide subsequent timestep allocation, rather than adding a correction-gradient term.
The original manuscript results provide additional evidence on the other tasks:
Original IS versus ISCV-SG results: three-seed mean accuracy (%), Tables 1, 2, and 9.
| Dataset | IS | ISCV-SG | Difference (percentage points) |
|---|---|---|---|
| GSM8K | 65.68 | 65.66 | −0.02 |
| OpenScience | 66.60 | 67.00 | +0.40 |
| MBPP | 41.25 | 42.15 | +0.91 |
Differences are calculated before rounding the displayed means. The near tie occurs on GSM8K, while OpenScience and MBPP show positive mean gains. These are the original manuscript comparisons; the additional statistic-only SG experiment was conducted on HiTab.
The new HiTab SG-Raw runs are independent of the historical HiTab IS runs under the same experimental protocol. Historical IS uses raw-loss statistics and has no CV term in its training scalar; both variants in the new SG comparison retain the detached CV term. The +2.23 pp gain across all three HiTab seeds thus provides direct training evidence for the value of confidence-corrected sampling statistics within SG.
Thank you to the authors for the detailed rebuttal and additional experiments.
I appreciate the new controlled experiment addressing W3, which provides convincing evidence for the contribution of confidence-corrected sampling statistics.
However, my main concerns regarding W1 and W2 remain. The performance improvements over strong baselines and the gains from online adaptation are still relatively marginal, particularly in light of the reported standard deviations. While I acknowledge the computational efficiency of the proposed method, I remain unconvinced that these modest improvements sufficiently establish its significance and novelty over existing timestep sampling approaches.
Therefore, I will maintain my original rating.
W1: Our main advantage is higher accuracy at essentially the same training time as Standard, and competitive accuracy at approximately half the training time of P-POTS+MIRROR. To address variation across seeds, we summarize the original manuscript results from Tables 1 and 9 as three-seed mean ± sample standard deviation. For each task, we select the highest-mean external baseline among Clipped, P-POTS, and P-POTS+MIRROR.
At comparable training time, ISCV-SG improves over Standard on all 12 same-seed comparisons across the four tasks with reported runtimes: GSM8K, MATH, HiTab, and OpenScience. The mean accuracy gains are +4.02, +2.65, +5.24, and +3.18 pp, respectively, with only 0.5–1.4% additional training time. These results show consistent accuracy gains with negligible training overhead.
At the matched token/update budgets used in the paper, ISCV-SG uses approximately 45–50% of P-POTS+MIRROR's measured training time on these four tasks, while achieving a higher mean accuracy on three of them. Across all five tasks, including MBPP, ISCV-SG has a higher mean than P-POTS+MIRROR on four of five tasks. Thus, P-POTS+MIRROR requires roughly twice the measured training time for comparable accuracy, demonstrating ISCV-SG's advantage in the accuracy–runtime trade-off.
Regarding hyperparameters, within each task, the compared methods use the same LoRA configuration and learning-rate schedule. For the ISCV controller, timestep binning and the default settings for estimating CV coefficients are shared across the reported LoRA experiments. The task-specific choices concern the correction-strength cap, adaptive sampling mass, and IS activation timing. Different dataset sizes and training horizons mean that the same absolute activation step can correspond to different training progress.
Appendix B.12 provides the following local seed-42 sensitivity measurements:
Increasing adaptive probability mass fourfold, from 5% to 20%, changes accuracy by 0.75 pp. Delaying IS activation from step 480 to 600 leaves the OpenScience score unchanged at 67.10%. The correction-cap sweep spans 1.74 pp. These measurements support local robustness over the tested settings, particularly for adaptive sampling mass and activation timing. The proposal-probability floor and importance-weight clipping did not activate in the all runs.
For different backbones, we evaluate LLaDA-8B with LoRA adaptation and 169M DiT with full-parameter pretraining. On the latter, the two-seed mean perplexity improves from 74.97 to 73.18 at 1B tokens, supporting applicability beyond LoRA adaptation on a single backbone.