- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Existing scaling laws for DLMs only focus on model capacity and data volume, allocating these uniformly across the diffusion process. However, the authors observe that DLM loss varies substantially with the diffusion timestep, so they expand the scaling law to a time- dependent setting. Ermine can better allocate training compute, and outperforms existing baselines.
Critical strengths
-
The research problem is novel and interesting. The paper shows that the loss improvement of DLMs is non-uniform and argues that a scaling law for DLMs should also answer at which timesteps compute should be spent. Prior DLM papers have not considered this dimension.
-
There is no need to tune a lot of hyperparameters, as these are automatically computed.
-
The theoretical derivation is clean and consistent with Chinchilla scaling law.
Critical weaknesses
I must state that I had no direct research experience on LLM scaling laws, so I may have missed something here.
In the paper, the authors assume l_t only depends on the resources at timestep t. Specifically, the authors claim: Empirically, however, we find that the time-wise losses are decoupled across timesteps, which allows M to remain the same as in Chinchilla. Specifically, the loss attained at one timestep is unaffected by the capacity and data allocated to the others, so each l can be fitted independently.
Although the authors provide some justification in the appendix, I believe it would be helpful to include a small-scale experiment that performs the exact fitting, and compares it with the approximation.
I would also refer to other reviewers' comments during the rebuttal stage.
Questions for authors
Could the authors add such a comparison (as mentioned before) to show how much the approximation deviates from the exact solution?
3 comments
Thanks! I appreciate the additional experiments and keep my positive score of 4.
Thank you for taking the time to read our response. We appreciate your thoughtful review.
We sincerely thank the reviewer for the constructive feedback.
Q1. In the paper, the authors assume
l_tonly depends on the resources at timestept. Specifically, the authors claim:Empirically, however, we find that the time-wise losses are decoupled across timesteps, which allows M to remain the same as in Chinchilla. Specifically, the loss attained at one timestep is unaffected by the capacity and data allocated to the others, so each l can be fitted independently.Although the authors provide some justification in the appendix, I believe it would be helpful to include a small-scale experiment that performs the exact fitting, and compares it with the approximation.Response. We thank the reviewer for raising this point. Exact fitting would require enumerating combinations of (Nt,Dt) across timesteps, which is computationally infeasible within the rebuttal period. To support our observation that time-wise losses can be estimated separately, we instead added two analyses in Appendix A.4.1: (A1) tests whether ℓt is sensitive to allocations at other timesteps, and (A2) tests whether Ermine's total-loss prediction matches measurement from 1018 to 1020 FLOPs.
(A1) We trained four variants with (Nt,Dt) fixed within the target region t∈[0.375,0.625] and with data or capacity densities shifted toward the clean or noisy end outside it (Fig. A1). Across all variants, the loss at t=0.5 remains close to the baseline with only 0.26-0.67% deviation, suggesting that cross-timestep effects are small.
(A2) We compared the predicted optimal loss L^∗=∫01ℓ^t(Nt∗,Dt∗)dt with the measured validation loss of MDM-MoE trained with the Ermine allocation (Fig. A2 and Table A7):
The prediction matches the measurement with a mean absolute error of 0.26% and a maximum of 0.95%. To conclude, (A1) and (A2) indicate that fitting each ℓ^t separately is an accurate approximation in practice.