- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on compute-optimal scaling for text-to-image diffusion model training. The authors find that earlier scaling studies use limited compute and provide insufficient guidance on how to allocate resources between model size and training data. They therefore propose Abra, a controlled family of flow-matching transformers ranging from 60 million to two billion parameters for systematic scaling analysis. To validate their method, they conduct several experiments, including scaling-law fitting, overtraining analysis, generation evaluation, linear probing, and resolution ablations. The experimental results demonstrate that loss-optimal training requires approximately 200 image tokens per parameter.
Critical strengths
-
This paper introduces a compute-optimal scaling study of text-to-image diffusion transformers and identifies an actionable rule of approximately 200 image tokens per parameter.
-
The study spans three orders of magnitude of compute, extending beyond the range of previous compute-optimal analyses for text-to-image models.
-
The authors extend the analysis beyond training loss to generative metrics, representation quality, and resolution, and further verify the main estimate using different fitting procedures.
Critical weaknesses
-
My main concern is that the paper presents 200 TPP as an actionable rule and claims that diffusion models "scale just as predictably as language models." However, all fits use models of at most 2B parameters, and no prediction is tested beyond this range. Section 4.4 even predicts a joint optimum for an untrained 6B model. The reliability of this rule for larger-scale training therefore remains unvalidated.
-
Section 4.1 derives iso-FLOP profiles from intermediate checkpoints of a single constant-LR run per model. Because the number of steps is fixed and batch size scales with model size, each TPP corresponds to nearly the same training step across model sizes. The fixed EMA decay therefore averages over a larger share of training at low TPP. Given the flat profiles, such step-dependent effects could shift the optimum, while the robustness check in Section 4.1 varies only the fitting procedure on the same checkpoints.
-
Section 4.2 measures the overtraining penalty as a percentage of total loss. Because total loss includes an irreducible floor that differs between the diffusion and LLM objectives, this normalization does not make the two directly comparable.
Questions for authors
-
Since Section 4.4 predicts a joint optimum for a 6B model, could the authors train this model and report whether its measured loss and probe accuracy match the fitted predictions?
-
Could the authors validate several iso-FLOP points with horizon-matched runs, for example using an LR cooldown or an EMA length scaled to the training duration? Please report whether the approximately 200 TPP optimum persists.
-
Could the authors also report the penalty relative to reducible loss, for example using the irreducible-loss estimate from Section 4.5, and show its sensitivity to plausible floor values?
3 comments
Thank you for the detailed response. The rebuttal addressed some of my concerns, and I understand that validating the scaling law at substantially larger model sizes would require considerable compute.
My concern is how much practical guidance the results offer for training today’s text-to-image models. I appreciate that Abra extends the controlled experiments to higher compute budgets than Liang et al. (2024), Scaling Laws for Diffusion Transformers, which already studied compute-optimal scaling for diffusion transformers. The empirical validation in Abra, however, goes up to 2B parameters. By the time Abra appeared, open-weight models at substantially larger scales had already been available since 2024, including Stable Diffusion 3.5 Large (8.1B) and FLUX.1 dev (12B). This gap limits how confidently practitioners can apply the reported compute-optimal allocation when training models at those scales.
My assessment of the paper is more positive after the rebuttal, and I would now consider it a borderline case. Since the rating scale does not offer a borderline option, I am keeping my original score. If the AC considers the empirical contribution sufficient within the tested range, I would be comfortable with acceptance.
Thank you for the updated comment and your engagement with us in this review process.
The reviewer is concerned that the lack of larger models limits the practical guidance and insufficiently differentiates our work from the previous work of Liang et al. (2024). However, the practical guidance that we provide is precisely what sets us apart. We produce a stronger fit and more accurate extrapolation than previous works. We note that Liang et al. do not report any quantitative information about their fits or their extrapolation. We validate extrapolation empirically in our paper: by treating the 2B model as an extrapolation target, we obtain a TPP of 195 instead of 199, with less than 0.002% relative error in loss for the extrapolation.
Extending our study to larger, potentially frontier-scale models is impractical. At current AWS H100 costs, and assuming an MFU of 30%, training a 6B model in our ladder would cost between \800,000. A 12B model would cost between \3,300,000. These are prices reserved for a stand-alone, frontier-scale training effort, of which a scaling law study is an essential sub-component.
Ultimately, the true practical value in a study like ours is guiding practitioners in their preparations for a large-scale run. Previous works provided insufficient guidance, but our work, through its controlled ladder and demonstrated extrapolation, closes this gap. Teams looking to train a frontier model can use our results to calibrate and sanity-check their own scaling analysis. Since every frontier effort should fit its own ladder, the 200 TPP compute-optimality finding gives them a strong starting point.
We appreciate that the assessment of the work is more positive after the initial rebuttal and that the reviewer is conditionally comfortable with acceptance. We hope that our work’s empirical contributions, clarified above, can be more directly reflected in the paper’s final assessment.
W1/Q1: Extrapolation beyond 2B. We empirically validated the quality of extrapolation for the largest model we trained. In Appendix G we study a scaling law fit only to the 120M-1B models and we find that with the 2B model included we get an optimal TPP of 199 and with it excluded we get an optimal TPP of 195, so the actionable ∼200 TPP rule holds in the range we study. The joint optimum statements in Section 4.4 and Appendix D are implied by extrapolating our fits but do not constitute an empirical result. We have revised both sections to make our intent more clear. Testing this prediction would require training a 6B model past compute optimality to 400 TPP (∼1023 FLOPs), 10x more compute than our 2B run and approaching the amount of compute used to pre-train GPT-3: ∼3×1023 FLOPs (Brown et al., 2020). This amount of compute is not feasible for our study.
W2/Q2: Constant LR and fixed EMA. Our choice of setup naturally follows from two requirements, and the requested horizon-matched runs would violate both. First, we want to match how diffusion models are trained in practice (Peebles & Xie, 2023; Li et al., 2024a). This means training with a flat learning rate and a fixed-timescale EMA. Our ~200 TPP optimum is therefore compute optimal for this standard approach. Second, because we must train every model to 400 TPP (vs. ~60-80 TPP for LLMs), we can afford only a single run for the largest model sizes. A flat learning rate and a fixed number of steps allow us to compare all models fairly as a function of steps, since steps control the EMA timescale. Thus, all models at 20 TPP, for example, have had the same number of EMA updates. Runs using learning rate decay are not standard practice in diffusion model training, and we would need to re-run every model at multiple terminal TPPs for a fair study. This amounts to re-running our entire study multiple times, realistically around 10 times per model size, with amortization putting us at 55x the compute of previous works. Finally, an EMA scaled to training duration requires knowing how EMA should scale with token horizon, which is a separate scaling axis from the one studied here. In LLMs, the analogous questions for learning rate and weight decay (Bergsma et al., 2025) have each necessitated dedicated studies, and for EMA in diffusion training there is no established rule to follow (Karras et al., 2024).
W3/Q3: Penalties relative to reducible loss. We recomputed the penalty relative to reducible loss and attach the figure in the supplementary material. Including reducible loss strengthens our conclusion, so we retain our existing analysis in the paper as the more conservative comparison. The figure shows the original analysis on the left, the version normalized by the fitted irreducible loss in the middle, and the p=95 worst-case fitted irreducible loss on the right. In all three cases our conclusions comparing the two model families hold, including when we compare the worst-case diffusion model to the best-case LLM.