acceptodds
Under review as a conference paper at ICLR 2027

Ermine: Neural Differential Scaling Laws for Diffusion Language Models

Abstract

Scaling laws prescribe how to spend compute; for diffusion language models (DLMs), they should also prescribe *when*. Existing compute-optimal recipes for DLMs inherit the Chinchilla scaling law, which allocates capacity and data uniformly across the diffusion process. We observe, however, that DLM loss varies substantially with the diffusion timestep, leaving compute underutilized under Chinchilla-style allocation. We introduce *Ermine*, a differential scaling law that characterizes the instantaneous loss via time-dependent capacity and data densities. We derive its optimal solution under a fixed compute budget and realize it with quantile-based stratified sampling and a sparse Mixture-of-Experts (MoE) architecture. Ermine accelerates Chinchilla-optimal training by , improves compute efficiency over dense DLMs by , and narrows the compute-optimal gap between DLMs and autoregressive models (ARMs) from to . On OpenWebText, our model attains a state-of-the-art validation perplexity of among DLMs, and at FLOPs it outperforms existing ARMs and DLMs on five zero-shot commonsense reasoning benchmarks using only -% of their training FLOPs. Our analysis further reveals that Ermine generalizes better by avoiding excessive compute in time regions where scalability diminishes.

Primary area:
generative models
Keywords:
Diffusion Language Models, Scaling Laws, Compute Allocation, LLM Pre-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_y3qvOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Existing scaling laws for DLMs only focus on model capacity and data volume, allocating these uniformly across the diffusion process. However, the authors observe that DLM loss varies substantially with the diffusion timestep, so they expand the scaling law to a time- dependent setting. Ermine can better allocate training compute, and outperforms existing baselines.

Critical strengths
  1. The research problem is novel and interesting. The paper shows that the loss improvement of DLMs is non-uniform and argues that a scaling law for DLMs should also answer at which timesteps compute should be spent. Prior DLM papers have not considered this dimension.

  2. There is no need to tune a lot of hyperparameters, as these are automatically computed.

  3. The theoretical derivation is clean and consistent with Chinchilla scaling law.

Critical weaknesses

I must state that I had no direct research experience on LLM scaling laws, so I may have missed something here.

In the paper, the authors assume l_t only depends on the resources at timestep t. Specifically, the authors claim: Empirically, however, we find that the time-wise losses are decoupled across timesteps, which allows M to remain the same as in Chinchilla. Specifically, the loss attained at one timestep is unaffected by the capacity and data allocated to the others, so each l can be fitted independently.

Although the authors provide some justification in the appendix, I believe it would be helpful to include a small-scale experiment that performs the exact fitting, and compares it with the approximation.

I would also refer to other reviewers' comments during the rebuttal stage.

Questions for authors

Could the authors add such a comparison (as mentioned before) to show how much the approximation deviates from the exact solution?

3 comments

We sincerely thank the reviewer for the constructive feedback.


Q1. In the paper, the authors assume l_t only depends on the resources at timestep t. Specifically, the authors claim: Empirically, however, we find that the time-wise losses are decoupled across timesteps, which allows M to remain the same as in Chinchilla. Specifically, the loss attained at one timestep is unaffected by the capacity and data allocated to the others, so each l can be fitted independently. Although the authors provide some justification in the appendix, I believe it would be helpful to include a small-scale experiment that performs the exact fitting, and compares it with the approximation.

Response. We thank the reviewer for raising this point. Exact fitting would require enumerating combinations of (Nt,Dt)(N_t, D_t) across timesteps, which is computationally infeasible within the rebuttal period. To support our observation that time-wise losses can be estimated separately, we instead added two analyses in Appendix A.4.1: (A1) tests whether ℓt\ell_t is sensitive to allocations at other timesteps, and (A2) tests whether Ermine's total-loss prediction matches measurement from 101810^{18} to 102010^{20} FLOPs.

(A1) We trained four variants with (Nt,Dt)(N_t, D_t) fixed within the target region t∈[0.375,0.625]t \in [0.375, 0.625] and with data or capacity densities shifted toward the clean or noisy end outside it (Fig. A1). Across all variants, the loss at t=0.5t = 0.5 remains close to the baseline with only 0.26-0.67% deviation, suggesting that cross-timestep effects are small.

(A2) We compared the predicted optimal loss L^∗=∫01ℓ^t(Nt∗,Dt∗)dt\hat{\mathcal{L}}^{\ast} = \int_0^1 \hat{\ell}_t(N_t^{\ast}, D_t^{\ast}) \mathrm{d}t with the measured validation loss of MDM-MoE trained with the Ermine allocation (Fig. A2 and Table A7):

FLOPsPredictedMeasuredError
1.00×10181.00 \times 10^{18}4.12014.1138+0.15%
2.15×10182.15 \times 10^{18}3.93603.9194+0.42%
4.64×10184.64 \times 10^{18}3.76753.7656+0.05%
1.00×10191.00 \times 10^{19}3.61323.6053+0.22%
2.15×10192.15 \times 10^{19}3.47193.4720+0.00%
4.64×10194.64 \times 10^{19}3.34233.3403+0.06%
1.00×10201.00 \times 10^{20}3.22333.1931+0.95%

The prediction matches the measurement with a mean absolute error of 0.26% and a maximum of 0.95%. To conclude, (A1) and (A2) indicate that fitting each ℓ^t\hat{\ell}_t separately is an accurate approximation in practice.

Thanks! I appreciate the additional experiments and keep my positive score of 4.

Thank you for taking the time to read our response. We appreciate your thoughtful review.

Reviewer_qDnvOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that compute-optimal recipes for diffusion language models (DLMs), inherited from the Chinchilla law, spread model capacity and data uniformly over diffusion time, although loss improvements with compute concentrate at intermediate timesteps. It proposes Ermine, which fits a Chinchilla-form law separately at each timestep in terms of a time-dependent parameter density and data density, and minimizes the integrated loss under a total-FLOP constraint. A Lagrangian argument gives a closed-form optimal compute density and its split into capacity and data, along with variants in which either model size or data is held constant across time. The allocation is realized with quantile-based timestep sampling and a mixture-of-experts (MoE) model whose number of active experts varies across timestep intervals. On C4, Ermine is reported to improve the compute efficiency of an MoE masked DLM by about 1.35× over the Chinchilla allocation.

Critical strengths
  1. Principled reformulation. Resolving the Chinchilla law along diffusion time turns hand-designed timestep densities and capacity schedules into a budget-dependent constrained optimization. The solution is cheap to compute, has a clear equal-marginal-return interpretation, and its fixed-capacity and fixed-data variants extend it to settings where only one quantity can vary.

  2. Systematic scaling study with interpretable regularities. Sweeps cover masked and uniform-state DLMs with dense and MoE backbones, plus an autoregressive reference. They show that compute is most valuable where masked and clean tokens are balanced, that this peak moves with the noise schedule in line with the log-SNR, and that scalability diminishes at the noisy end. Showing that the standard OpenWebText setup is far from compute-optimal is useful in itself.

Critical weaknesses
  1. Unclear gain over simple rules. Away from the noisy end, prescribed data densities are broad and fairly flat, and the best hand-set Gaussian is within two standard deviations of Ermine (AS4). A truncated-uniform baseline, and reusing one budget's allocation at another, would show what the fitted shape adds.

  2. Overstated downstream and generalization claims. Ermine beats external models only on average, trailing several on HellaSwag and PIQA, and they differ in data and architecture. Held-out gains peak at the noisiest timestep and reverse at the cleanest; the main variant loses to the Chinchilla allocation on three of seven zero-shot sets. Please temper these claims.

Questions for authors

Were the coefficients refit for the OpenWebText (DiT) and Dolma (OLMoE-style) settings, or transferred from the C4 fits, which cover only smaller budgets? What losses does the law predict there for each allocation, compared with the measured ones?

5 comments

We sincerely thank the reviewer for the constructive feedback.


Q1.
(a) Unclear gain over simple rules. Away from the noisy end, prescribed data densities are broad and fairly flat, and the best hand-set Gaussian is within two standard deviations of Ermine (AS4).
(b) A truncated-uniform baseline, and reusing one budget's allocation at another, would show what the fitted shape adds.

Response.
(a) In Table A11, although Ermine (Time-Invariant NN) falls within two standard deviations of the best Gaussian baseline (μ=0.5\mu = 0.5, σ=0.1\sigma = 0.1; 3.626±0.0103.626 \pm 0.010), the full Ermine allocation achieves 3.599±0.0043.599 \pm 0.004, establishing a gap of 0.0270.027 (about 2.5 combined standard deviations). Moreover, the Gaussian baseline requires a careful hyperparameter search. The other two (μ,σ)(\mu, \sigma) settings (3.7073.707 and 3.7093.709) even underperform the Chinchilla baseline (3.6753.675), whereas Ermine derives its allocation directly from the fitted law.

(b) We thank the reviewer for this suggestion. To further examine the benefits of the fitted allocation, we have extended the ablation study in Appendix A.6.1 with two additional baselines:

  • (i) Truncated-uniform allocation. We compare Ermine (Time-invariant NN) against Chinchilla with truncated-uniform sampling over three intervals (ts,te)∈{(0.0,0.5),(0.3,0.8),(0.5,1.0)}(t_s, t_e)\in\{(0.0,0.5),(0.3,0.8),(0.5,1.0)\} and with hyperparameter selection following [1], under the same setup as AS4. These baselines perform worse than the best Gaussian allocation:
AllocationLoss
Ermine (Time-Invariant NN)3.608$\pm$0.008
Chinchilla3.675$\pm$0.016
Chinchilla + Gaussian (μ\mu = 0.5, σ\sigma = 0.1)3.626$\pm$0.010
Chinchilla + Trunc. Uniform DtD_t (tst_s = 0.3, tet_e = 0.8)3.631$\pm$0.009
Chinchilla + Trunc. Uniform DtD_t (tst_s = 0.0, tet_e = 0.5)3.717$\pm$0.013
Chinchilla + Trunc. Uniform DtD_t (tst_s = 0.5, tet_e = 1.0)3.917$\pm$0.026
  • (ii) Budget-fixed allocation. We derive the Ermine allocation at (C=1018C=10^{18}) and (5×10185\times10^{18}) FLOPs and transfer them to a C=1019C=10^{19}-FLOPs run. We found that the original prediction with C=1019C=10^{19} yields the best loss.
AllocationLoss
Ermine (C=1×1019C=1\times 10^{19})3.599$\pm$0.004
Ermine (C=5×1018C=5\times 10^{18})3.604$\pm$0.009
Ermine (C=1×1018C=1\times10^{18})3.609$\pm$0.009

Reference
[1] Arriola et al., Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR 2025.


Q2. Overstated downstream and generalization claims.
(a) Ermine beats external models only on average, trailing several on HellaSwag and PIQA, and they differ in data and architecture.
(b) Held-out gains peak at the noisiest timestep and reverse at the cleanest;
(c) the main variant loses to the Chinchilla allocation on three of seven zero-shot sets.
Please temper these claims.

Response. We thank the reviewer for this suggestion and have tempered all three claims.

(a) We have revised the Abstract and the corresponding contribution in the Introduction to "outperforms existing ARMs and DLMs in average accuracy on five benchmarks."

(b) We have revised Section 4.4 to state that "...Ermine improves not only training efficiency but also zero-shot generalization at most timesteps…" acknowledging the slight reversal at the cleanest timesteps.

(c) We have retitled the subsection in Appendix A.6.3 from "(S1) Ermine Outperforms Chinchilla on Zero-shot Perplexity Evaluation" to "(S1) Zero-shot Perplexity Evaluation". We have also revised the text to report that Ermine improves over Chinchilla on four of seven datasets, while trailing it on LAMBADA, ArXiv, and PubMed.

Q3.
(a) Were the coefficients refit for the OpenWebText (DiT) and Dolma (OLMoE-style) settings, or transferred from the C4 fits, which cover only smaller budgets?
(b) What losses does the law predict there for each allocation, compared with the measured ones?

Response. Thanks for raising this insightful question.

(a) The coefficients were fitted on C4 and transferred to the OpenWebText (OWT) and Dolma settings without refitting. This follows common practice in transferring compute-optimal allocations across datasets; for example, Hoffmann et al. find consistent allocations on C4 and GitHub (see Appendix C, Fig. A2 in [1]) . We note that only the allocation is transferred. The absolute loss depends largely on the irreducible term, which reflects the entropy of the data distribution and varies with the dataset, tokenizer, and sequence length, so absolute predicted losses are not expected to transfer.

(b) Because the data entropy differs across datasets, the predicted absolute losses are expected to deviate from the measured ones. For completeness, the table below reports the losses predicted by the C4-fitted law alongside the measured losses.

OWT (C=2.89×1020C=2.89\times 10^{20})Dolma (C=1.1×1021C=1.1\times 10^{21})
Predicted Loss3.0902.914
Measured Loss2.8702.983
Error+7.7%-2.3%

Reference
[1] Hoffmann et al. Training Compute-Optimal Large Language Models. 2022.

Please let us know if you have any follow-up questions on our response. We will be glad to clarify before the discussion closes on Nov. 18.

Thank you!

Reviewer_rkoMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

My understanding is that this paper examines how to allocate a fixed training compute budget across diffusion timesteps in terms of model capacity and training data for Diffusion Language Models (DLMs). The authors observe that loss reductions from additional compute vary across timesteps, whereas conventional Chinchilla-based recipes allocate resources uniformly over time. They propose Ermine, a loss scaling law for timestep-dependent capacity and data, and derive an optimal allocation under this model for a fixed compute budget. They implement the allocation through quantile-based timestep sampling and a Mixture-of-Experts (MoE) architecture that varies the number of active experts with the timestep. The reported results include a 1.35-fold improvement in compute efficiency over Chinchilla-optimal allocation, a perplexity upper bound of 17.63 on OpenWebText, and improved average accuracy across five zero-shot reasoning tasks. The paper also examines generalization to unseen domains and attributes the observed improvements to avoiding excessive compute allocation to timesteps with limited scalability.

Critical strengths

S1. I consider the joint allocation of capacity and data across timesteps a methodological strength. The paper models timestep-wise loss as a function of both quantities (Eq. (6)), derives their allocation under a fixed compute budget (Eqs. (8)-(10)), and describes how to implement it through timestep sampling and variable top-k MoE routing (Section 3.2). The timestep-wise scaling analysis is thus used to choose both the sampling distribution and the active capacity during training.

S2. I find the matched Dolma comparison convincing because it holds the backbone, data, and compute budget fixed. At 1.1e21 FLOPs, the reported average accuracy across five tasks increases from 57.04 to 60.01 (Table 3, "Aligned Models"). Figure 7 also compares dense MDM, MoE with Chinchilla allocation, and MoE with Ermine allocation, reporting compute-efficiency improvements of 1.36x from MoE and a further 1.35x from Ermine. I take these comparisons as evidence that changing the allocation helps beyond introducing MoE, although they do not by themselves establish optimality or explain why the gains occur.

Critical weaknesses

W1. If I understand correctly, Eq. (6) predicts each timestep's loss from the capacity and data allocated to that timestep. In the MoE implementation (Section 3.2), however, updates to shared parameters could also affect losses at other timesteps. Table A5 provides supporting prediction results for composed models, but I could not determine whether it tests changes to allocations at other timesteps while keeping the target timestep's capacity and data fixed. I would like the authors to clarify the parameter-sharing structure and allocation conditions used in that validation, since these determine how directly it supports the loss model used for joint allocation.

W2. I could not determine how accurately the fitted loss law predicts evaluation loss after the timestep sampling distribution is changed. Eqs. (6) and (8) use the predicted timestep-wise losses to derive the allocation, while Section 3.2 realizes the data allocation by replacing uniform sampling intervals with quantile-based intervals. Figure 7 and Table 3 show that the resulting allocation improves performance, but they do not directly compare predicted and measured losses. I would like the authors to specify the sampling conditions used for fitting and clarify whether the prediction results in Table A5 also cover the quantile-based sampling used by Ermine.

Questions for authors

Q1. This paper seems to have several citation errors, but I'm not exactly sure. Could the authors check all the citations? Please revise them if any are incorrect.

Q2. In Section 2.1, the text states that "a is the slope," while the displayed expression is log(A N^{-a}) = log A - a log N. Should the slope be described as -a, or a as the magnitude of the negative slope?

Q3. Proposition 1 describes the optimal solution as minimizing Eq. (9), which is the Lagrangian. Would it be more precise to say that the solution minimizes the constrained objective in Eq. (8)? My understanding is that the multiplier is determined by the budget constraint, rather than jointly minimized with the allocation variables.

Q4. In Section 3.2, N_i^* first denotes the interval average of the continuous optimal capacity and is then equated to N_0 + N_Delta k_i. Since k_i must be an integer, could you clarify when this equality holds exactly and distinguish the target capacity from the realized capacity if rounding is needed?

Q5. In Section 3.2, the boundaries are defined as tau_0 = 0 and tau_u = 1. However, Figure 6 appears to place tau_0 at t = 1 and tau_u at t = 0. Could you check the boundary labels and the direction of the CDF? The plotted cumulative values appear to increase as t decreases.

Q6. The Abstract and Introduction report a 1.87x improvement in compute efficiency over dense DLMs, whereas Figure 7 reports gains of 1.36x from MoE and a further 1.35x from Ermine allocation. Could you explain how 1.87x is calculated? If these ratios are measured at different target losses, please specify those evaluation points, or let me know if I'm misunderstanding.

Q7. The Introduction describes the OWT result of 17.63 as "approaching the ARM result (17.54)", while Table 2 reports 12.98 for the compute-optimal ARM at the same budget. Could you clarify whether 17.54 refers to the earlier ARM configuration rather than the compute-optimal comparison?

Q8. The Abstract states that the model "outperforms existing ARMs and DLMs on five" benchmarks. Did you mean that it achieves the highest average accuracy across the five tasks among the models in the first two blocks of Table 3? The table shows lower HellaSwag and PIQA scores than several reference models, while Section 4.3 distinguishes the average result from the individual task results.

Q9. Table 2 presents the DLM results as perplexity upper bounds, but Section 4.2 describes the values as perplexities without this qualification. Could you make the upper-bound interpretation explicit in the text as well? This would help distinguish the reported ELBO-based comparison from a comparison of exact perplexities.

9 comments

We sincerely thank the reviewer for the constructive feedback.


W1.
(a) If I understand correctly, Eq. (6) predicts each timestep's loss from the capacity and data allocated to that timestep. In the MoE implementation (Section 3.2), however, updates to shared parameters could also affect losses at other timesteps.

(b) Table A5 provides supporting prediction results for composed models, but I could not determine whether it tests changes to allocations at other timesteps while keeping the target timestep's capacity and data fixed.

(c) I would like the authors to clarify the parameter-sharing structure and allocation conditions used in that validation, since these determine how directly it supports the loss model used for joint allocation.

Response. We thank the reviewer for this comment.

(a) Yes. Eq. (6) models the time-wise loss as a function of the capacity and data allocated to that timestep. Updates to shared parameters in the MoE implementation could also affect the empirical losses at other timesteps. However, we observe that this influence is small, as shown by the controlled experiments described in (b) below and in our response to W2(a).

(b) To test whether the loss at a target timestep is affected by allocations at other timesteps, we added a controlled experiment in Appendix A.4.1 (denoted as (A1)). We trained four additional variants with (Nt,Dt)(N_t,D_t) fixed in the target region t∈[0.375,0.625]t\in[0.375,0.625], while varying allocations outside it: data and capacity densities concentrated toward the clean or noisy end (see Fig. A1). Across all variants, the loss at t=0.5t=0.5 deviates from the baseline by only 0.26%-0.67%, suggesting that cross-timestep allocation effects are small and supporting the approximation underlying Eq. (6).

(c) In our MoE implementation, all parameters (attention, router, and experts) are shared across timesteps; only the number of activated experts kik_i varies. Table A5 evaluates a model trained under Ermine's capacity and data allocations, using time-dependent kik_i for t∈(τi,τi−1)t\in (\tau_{i}, \tau_{i-1}) during evaluation.


W2.
(a) I could not determine how accurately the fitted loss law predicts evaluation loss after the timestep sampling distribution is changed. Eqs. (6) and (8) use the predicted timestep-wise losses to derive the allocation, while Section 3.2 realizes the data allocation by replacing uniform sampling intervals with quantile-based intervals. Figure 7 and Table 3 show that the resulting allocation improves performance, but they do not directly compare predicted and measured losses.

(b) I would like the authors to specify the sampling conditions used for fitting and clarify whether the prediction results in Table A5 also cover the quantile-based sampling used by Ermine.

Response. We thank the reviewer for raising these questions. Based on the response in W1, we have verified that the time-wise loss ℓ^t\hat{\ell}_t attained at one timestep is approximately independent of the capacity and data allocated to the others, so each ℓ^t\hat{\ell}_t can be fitted separately on runs with uniform timestep sampling.

(a) To examine gap between the predicted and measured total losses, we added a direct comparison in Appendix A.4.1 (denoted as (A2); Fig. A2 and Table A7) and summarize the result in the following table.

FLOPsPredictedMeasuredError
1.00×10181.00 \times 10^{18}4.12014.1138+0.15%
2.15×10182.15 \times 10^{18}3.93603.9194+0.42%
4.64×10184.64 \times 10^{18}3.76753.7656+0.05%
1.00×10191.00 \times 10^{19}3.61323.6053+0.22%
2.15×10192.15 \times 10^{19}3.47193.4720+0.00%
4.64×10194.64 \times 10^{19}3.34233.3403+0.06%
1.00×10201.00 \times 10^{20}3.22333.1931+0.95%

For MDM-MoE trained with the Ermine allocation at seven budgets from 101810^{18} to 102010^{20} FLOPs, the predicted optimal loss L^∗=∫01ℓ^t(Nt∗,Dt∗) dt\hat{\mathcal{L}}^{\ast} = \int_{0}^{1} \hat{\ell}_t(N_t^{\ast}, D_t^{\ast})\,dt matches the measured validation loss with a mean absolute error of 0.26% and a maximum of 0.95%.

(b) The coefficients were fitted on runs with uniform timestep sampling, and the prediction results in Table A5 were derived based on these coefficients. We have added this detail to Appendix A.4.1.

We have aggregated W1 and W2 as a unified discussion in Appendix A.4.1.

Q1. This paper seems to have several citation errors, but I'm not exactly sure. Could the authors check all the citations? Please revise them if any are incorrect.

Response. We have re-checked all citations and corrected three errors or inconsistencies we found:

  • (1) GPT-Neo in Table 3 previously cited (Black et al., 2022) and now cites (Black et al., 2021).
  • (2) Ludziejewski et al. (2025) and Muennighoff et al. (2025) are jointly cited in Sections 4.1, A.4.2, and A.5.1 for consistency.
  • (3) (von Rütte et al., 2025) is corrected as (Rütte et al., 2025) in Section A.3.2.

Please let us know if you have identified any other errors, and we will correct them.


Q2. In Section 2.1, the text states that "a is the slope," while the displayed expression is log(A N^{-a}) = log A - a log N. Should the slope be described as -a, or a as the magnitude of the negative slope?

Response. We thank the reviewer for identifying this wording error. We have changed the description “the slope” to “the magnitude of the slope” in Section 2.1 and Table A1.


Q3. Proposition 1 describes the optimal solution as minimizing Eq. (9), which is the Lagrangian.
(a) Would it be more precise to say that the solution minimizes the constrained objective in Eq. (8)?
(b) My understanding is that the multiplier is determined by the budget constraint, rather than jointly minimized with the allocation variables.

Response. We thank the reviewer for your careful reading.
(a) Yes, it is precise to say that the solution minimizes L^\hat{\mathcal{L}} in Eq. (8). We have adjusted the wording in Proposition 1 accordingly.
(b) Yes, the multiplier is not minimized over. It is uniquely determined by the budget constraint.


Q4. In Section 3.2, N_i^* first denotes the interval average of the continuous optimal capacity and is then equated to N_0 + N_Delta k_i. Since k_i must be an integer, could you clarify when this equality holds exactly and distinguish the target capacity from the realized capacity if rounding is needed?

Response. Yes, rounding is needed and we have revised Section 3.2 to distinguish the target capacity from the realized capacity.

Let N^i∗≜1τi−1−τi∫τiτi−1Nt∗dt\hat{N}_{i}^{\ast} \triangleq \frac{1}{\tau_{i-1}-\tau_{i}} \int_{\tau_{i}}^{\tau_{i-1}} N_{t}^{\ast} \mathrm{d}t be the target capacity. The equality holds exactly only when (N^i∗−N0)/NΔ(\hat{N}_{i}^{\ast} - N_0)/N_{\Delta} is an integer, which depends on the model configuration (N0N_0, NΔN_{\Delta}) and the partition (τi\tau_i). In practice, we use k^i=round((N^i∗−N0)/NΔ)\hat{k}_i = \mathrm{round}\big((\hat{N}_{i}^{\ast} - N_0)/N_{\Delta}\big), which yields the realized capacity N0+NΔk^iN_0 + N_{\Delta} \hat{k}_i.


Q5. In Section 3.2, the boundaries are defined as tau_0 = 0 and tau_u = 1. However, Figure 6 appears to place tau_0 at t = 1 and tau_u at t = 0. Could you check the boundary labels and the direction of the CDF? The plotted cumulative values appear to increase as t decreases.

Response. We thank the reviewer for catching this notation inconsistency. We have corrected the notation in Section 3.2 with Fig. 6 by setting τ0=1\tau_0 = 1 and τu=0\tau_u = 0 in the revised manuscript.

Q6. The Abstract and Introduction report a 1.87x improvement in compute efficiency over dense DLMs, whereas Figure 7 reports gains of 1.36x from MoE and a further 1.35x from Ermine allocation. Could you explain how 1.87x is calculated? If these ratios are measured at different target losses, please specify those evaluation points, or let me know if I'm misunderstanding.

Response. We thank the reviewer for pointing this out. The efficiency gains vary slightly with the target loss (from 1.83x to 1.91x), as shown in the table below (also added to Table A14). We have unified all reported ratios at a reference loss of 3.9 and revised the combined improvement in the Abstract and Introduction from 1.87x to 1.84x.

Validation lossMDM →\to MDM-MoE (Chinchilla)MDM-MoE (Chinchilla) →\to MDM-MoE (Ermine)MDM →\to MDM-MoE (Ermine)
4.01.35×1.35×1.83×
3.91.36×1.35×1.84×
3.81.37×1.34×1.85×
3.71.38×1.34×1.86×
3.61.39×1.34×1.87×
3.51.40×1.34×1.88×
3.41.42×1.34×1.89×
3.31.43×1.34×1.91×

Q7. The Introduction describes the OWT result of 17.63 as "approaching the ARM result (17.54)", while Table 2 reports 12.98 for the compute-optimal ARM at the same budget. Could you clarify whether 17.54 refers to the earlier ARM configuration rather than the compute-optimal comparison?

Response. Yes, 17.54 refers to the ARM result reported by Sahoo et al. (2024) under the conventional configuration (NN = 92M, DD = 524B), not to the compute-optimal ARM. We have revised the sentence to "approaching the ARM result (17.54) reported by Sahoo et al. (2024)" to avoid confusion.


Q8. The Abstract states that the model "outperforms existing ARMs and DLMs on five" benchmarks. Did you mean that it achieves the highest average accuracy across the five tasks among the models in the first two blocks of Table 3? The table shows lower HellaSwag and PIQA scores than several reference models, while Section 4.3 distinguishes the average result from the individual task results.

Response. Yes, our model achieves the highest average accuracy across the five tasks among the models in the first two blocks of Table 3 (60.01 vs. 59.42 for the strongest baseline). We have revised the Abstract to “outperforms existing ARMs and DLMs in average accuracy on five benchmarks" and made the same correction to the corresponding contribution in the Introduction.


Q9. Table 2 presents the DLM results as perplexity upper bounds, but Section 4.2 describes the values as perplexities without this qualification. Could you make the upper-bound interpretation explicit in the text as well? This would help distinguish the reported ELBO-based comparison from a comparison of exact perplexities.

Response. We thank the reviewer for this suggestion. We have revised Section 4.2 to state explicitly that the DLM values are perplexity upper bounds derived from the ELBO. We made the same clarification in the caption of Table A13.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.