acceptodds
Under review as a conference paper at ICLR 2027

Abra: Scaling Diffusion Image Training

Abstract

Compute-optimal scaling laws guide the training of frontier language models yet remain largely unexplored for visual generation. We present a systematic scaling law study for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained across three orders of magnitude worth of compute ( to FLOPs), reaching significantly larger compute budgets than previous works. We demonstrate that diffusion models scale just as predictably as language models but require far more data to train optimally: compute optimality occurs at approximately image tokens per parameter, ten times the Chinchilla compute-optimal prescription for LLMs. We show that unlike language models, diffusion models are robust to overtraining and that practitioners should err on the side of more data rather than a larger model. Finally, we show that this predictability extends beyond training loss to generative quality metrics, optimal CFG settings, representation quality, and even the shape of the training curves, which collapse onto a universal form.

Primary area:
generative models
Keywords:
diffusion, scaling laws, compute optimal, scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A6TaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on compute-optimal scaling for text-to-image diffusion model training. The authors find that earlier scaling studies use limited compute and provide insufficient guidance on how to allocate resources between model size and training data. They therefore propose Abra, a controlled family of flow-matching transformers ranging from 60 million to two billion parameters for systematic scaling analysis. To validate their method, they conduct several experiments, including scaling-law fitting, overtraining analysis, generation evaluation, linear probing, and resolution ablations. The experimental results demonstrate that loss-optimal training requires approximately 200 image tokens per parameter.

Critical strengths
  1. This paper introduces a compute-optimal scaling study of text-to-image diffusion transformers and identifies an actionable rule of approximately 200 image tokens per parameter.

  2. The study spans three orders of magnitude of compute, extending beyond the range of previous compute-optimal analyses for text-to-image models.

  3. The authors extend the analysis beyond training loss to generative metrics, representation quality, and resolution, and further verify the main estimate using different fitting procedures.

Critical weaknesses
  1. My main concern is that the paper presents 200 TPP as an actionable rule and claims that diffusion models "scale just as predictably as language models." However, all fits use models of at most 2B parameters, and no prediction is tested beyond this range. Section 4.4 even predicts a joint optimum for an untrained 6B model. The reliability of this rule for larger-scale training therefore remains unvalidated.

  2. Section 4.1 derives iso-FLOP profiles from intermediate checkpoints of a single constant-LR run per model. Because the number of steps is fixed and batch size scales with model size, each TPP corresponds to nearly the same training step across model sizes. The fixed EMA decay therefore averages over a larger share of training at low TPP. Given the flat profiles, such step-dependent effects could shift the optimum, while the robustness check in Section 4.1 varies only the fitting procedure on the same checkpoints.

  3. Section 4.2 measures the overtraining penalty as a percentage of total loss. Because total loss includes an irreducible floor that differs between the diffusion and LLM objectives, this normalization does not make the two directly comparable.

Questions for authors
  1. Since Section 4.4 predicts a joint optimum for a 6B model, could the authors train this model and report whether its measured loss and probe accuracy match the fitted predictions?

  2. Could the authors validate several iso-FLOP points with horizon-matched runs, for example using an LR cooldown or an EMA length scaled to the training duration? Please report whether the approximately 200 TPP optimum persists.

  3. Could the authors also report the penalty relative to reducible loss, for example using the irreducible-loss estimate from Section 4.5, and show its sensitivity to plausible floor values?

3 comments

W1/Q1: Extrapolation beyond 2B. We empirically validated the quality of extrapolation for the largest model we trained. In Appendix G we study a scaling law fit only to the 120M-1B models and we find that with the 2B model included we get an optimal TPP of 199 and with it excluded we get an optimal TPP of 195, so the actionable ∼200\sim 200 TPP rule holds in the range we study. The joint optimum statements in Section 4.4 and Appendix D are implied by extrapolating our fits but do not constitute an empirical result. We have revised both sections to make our intent more clear. Testing this prediction would require training a 6B model past compute optimality to 400 TPP (∼1023\sim 10^{23} FLOPs), 10x more compute than our 2B run and approaching the amount of compute used to pre-train GPT-3: ∼3×1023\sim 3\times 10^{23} FLOPs (Brown et al., 2020). This amount of compute is not feasible for our study.

W2/Q2: Constant LR and fixed EMA. Our choice of setup naturally follows from two requirements, and the requested horizon-matched runs would violate both. First, we want to match how diffusion models are trained in practice (Peebles & Xie, 2023; Li et al., 2024a). This means training with a flat learning rate and a fixed-timescale EMA. Our ~200 TPP optimum is therefore compute optimal for this standard approach. Second, because we must train every model to 400 TPP (vs. ~60-80 TPP for LLMs), we can afford only a single run for the largest model sizes. A flat learning rate and a fixed number of steps allow us to compare all models fairly as a function of steps, since steps control the EMA timescale. Thus, all models at 20 TPP, for example, have had the same number of EMA updates. Runs using learning rate decay are not standard practice in diffusion model training, and we would need to re-run every model at multiple terminal TPPs for a fair study. This amounts to re-running our entire study multiple times, realistically around 10 times per model size, with amortization putting us at 55x the compute of previous works. Finally, an EMA scaled to training duration requires knowing how EMA should scale with token horizon, which is a separate scaling axis from the one studied here. In LLMs, the analogous questions for learning rate and weight decay (Bergsma et al., 2025) have each necessitated dedicated studies, and for EMA in diffusion training there is no established rule to follow (Karras et al., 2024).

W3/Q3: Penalties relative to reducible loss. We recomputed the penalty relative to reducible loss and attach the figure in the supplementary material. Including reducible loss strengthens our conclusion, so we retain our existing analysis in the paper as the more conservative comparison. The figure shows the original analysis on the left, the version normalized by the fitted irreducible loss in the middle, and the p=95 worst-case fitted irreducible loss on the right. In all three cases our conclusions comparing the two model families hold, including when we compare the worst-case diffusion model to the best-case LLM.

Thank you for the detailed response. The rebuttal addressed some of my concerns, and I understand that validating the scaling law at substantially larger model sizes would require considerable compute.

My concern is how much practical guidance the results offer for training today’s text-to-image models. I appreciate that Abra extends the controlled experiments to higher compute budgets than Liang et al. (2024), Scaling Laws for Diffusion Transformers, which already studied compute-optimal scaling for diffusion transformers. The empirical validation in Abra, however, goes up to 2B parameters. By the time Abra appeared, open-weight models at substantially larger scales had already been available since 2024, including Stable Diffusion 3.5 Large (8.1B) and FLUX.1 dev (12B). This gap limits how confidently practitioners can apply the reported compute-optimal allocation when training models at those scales.

My assessment of the paper is more positive after the rebuttal, and I would now consider it a borderline case. Since the rating scale does not offer a borderline option, I am keeping my original score. If the AC considers the empirical contribution sufficient within the tested range, I would be comfortable with acceptance.

Thank you for the updated comment and your engagement with us in this review process.

The reviewer is concerned that the lack of larger models limits the practical guidance and insufficiently differentiates our work from the previous work of Liang et al. (2024). However, the practical guidance that we provide is precisely what sets us apart. We produce a stronger fit and more accurate extrapolation than previous works. We note that Liang et al. do not report any quantitative information about their fits or their extrapolation. We validate extrapolation empirically in our paper: by treating the 2B model as an extrapolation target, we obtain a TPP of 195 instead of 199, with less than 0.002% relative error in loss for the extrapolation.

Extending our study to larger, potentially frontier-scale models is impractical. At current AWS H100 costs, and assuming an MFU of 30%, training a 6B model in our ladder would cost between \400,000and400,000 and \\800,000. A 12B model would cost between \1,650,000and1,650,000 and \\3,300,000. These are prices reserved for a stand-alone, frontier-scale training effort, of which a scaling law study is an essential sub-component.

Ultimately, the true practical value in a study like ours is guiding practitioners in their preparations for a large-scale run. Previous works provided insufficient guidance, but our work, through its controlled ladder and demonstrated extrapolation, closes this gap. Teams looking to train a frontier model can use our results to calibrate and sanity-check their own scaling analysis. Since every frontier effort should fit its own ladder, the 200 TPP compute-optimality finding gives them a strong starting point.

We appreciate that the assessment of the work is more positive after the initial rebuttal and that the reviewer is conditionally comfortable with acceptance. We hope that our work’s empirical contributions, clarified above, can be more directly reflected in the paper’s final assessment.

Reviewer_egsCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a systematic study of compute-optimal scaling laws for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained over a wide range of compute budgets. The authors find that diffusion models exhibit predictable scaling behavior similar to language models, but require substantially more data, with compute-optimal training occurring at roughly 200 image tokens per parameter. They further show that diffusion models are relatively robust to overtraining and that allocating more compute to data is generally preferable to simply increasing model size. Beyond training loss, the observed scaling behavior also extends to generative quality, optimal CFG settings, representation quality, and the shape of training curves.

Critical strengths
  1. The problem is practically important. The paper studies how to allocate model size and training data under a fixed compute budget, providing useful guidance for planning the training of text-to-image models.

  2. The experiments cover a relatively broad scale. The model size ranges from 60M to 2B parameters, with compute budgets spanning three orders of magnitude. The study also analyzes training loss, generation quality, and representation quality.

  3. The findings are insightful. In addition to identifying an empirical rule of approximately 200 tokens per parameter, the paper shows that different evaluation metrics may favor different model-data allocations, highlighting the limitations of relying solely on training loss for model selection.

Critical weaknesses
  1. The generalizability of the conclusions is limited. The experiments are conducted primarily with a single model family, dataset, and fixed VAE and text encoder. It remains unclear whether the 200-TPP rule holds under other model architectures and training setups.

  2. The comparison with language models should be interpreted with caution. Image tokens and text tokens differ substantially in information density, so the claim that diffusion models require “10× more data” may overstate the direct comparability between the two modalities.

  3. Some conclusions rely on strong assumptions or extrapolation. The resolution-scaling experiments assume scaling exponents of 0.5 for both model size and data, while some predictions for larger models extend beyond the range that is directly validated experimentally.

Questions for authors

Refer to weakness.

2 comments

W1: Generalizability. We don’t yet know how the 200 TPP value would be affected by a different VAE, model family, or dataset. Answering these important questions was out of scope of this work as it would require another full scaling ladder for each of these ablative axes. We followed the standard practice of compute-optimal scaling studies and fixed many components in order to study the core scaling behavior. If the LLM scaling law literature is any indication, we expect that reasonable changes to the architecture will not substantially change the results, but we do not have sufficient information to make any conjecture regarding the VAE or dataset choices. We would very much like to know the answers to these questions; see our response to Reviewer 3SAA for further discussion.

W2: Comparison with LLMs. We agree that text and image tokens differ in their information density and this is likely one of several causal factors leading to higher optimal TPP for diffusion models. Throughout the paper D is measured in tokens following prior works (Kaplan et al., 2020; Hoffmann et al., 2022; Liang et al., 2024), and 10x more data refers to needing 10 times as many tokens to reach compute optimality at a fixed model size for a diffusion model compared to an LLM.

W3: Extrapolation. Regarding the resolution-scaling experiments, we chose to use 0.5 as the exponent since fitting only three models leads to looser fits. This exponent is derived from the much stronger fits performed on 512x512 images (Figure 1). Regarding the extrapolation, the reviewer is highlighting statements we made in Section 4.4 and Appendix D. Upon re-reading these statements we agree that our intention is not sufficiently clear. We intended these statements to be a “napkin math conjecture”, but to be clear, we do not have sufficient empirical evidence to state that this crossover does happen. We have edited the language in these sections to make our intention more clear.

We do empirically validate the extrapolation of our fits in Appendix G (see Tables 4, 5 and the surrounding text) where we use the 2B model as a validation point against the law fit only to models through 1B. With the 2B model removed we find an optimal TPP of 195 vs. 199 with it included.

Thank you for the detailed response. I appreciate the authors' efforts to address my concerns. I will keep my positive rating.

Reviewer_3SAAOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically studies compute-optimal scaling laws for text-to-image flow models from 60M to 2B parameters. It finds that diffusion loss is optimized at ~200 image tokens per parameter, with little loss penalty from training smaller models for longer. Generative metrics and representation quality also scale predictably, but favor different allocations between model size and training data.

Critical strengths
  • It is very valuable to conduct such a controlled scaling study over a substantial compute range for flow models, providing actionable insights to train industrial-level models. This paper studies resource allocation from 60M to 2B parameters, with muP-based HP transfer, EMA-model evaluation, and iso-FLOP analysis. The ~200 TPP optimum for diffusion loss and the relatively flat penalty on the overtraining side provide concrete insights and constitute the paper's strong contribution.

  • Besides, the paper provides findings that different evaluation objectives (such as FID/KID, CLIPScore/CMMD, and linear-probe performance) favor different allocations between model capacity and training data, which further strengthens its practical relevance.

Critical weaknesses

None

Questions for authors

I'm interested in how much the ~200 TPP compute-optimal point depends on the training configuration. In particular, could the timestep schedule, caption information density (e.g., short, long, or structured json captions), or model architecture (e.g., single-stream DiT, MMDiT, or MoE) substantially change these findings? I understand that systematically studying these factors would require large compute and am not requesting a comprehensive experimental sweep. Rather, based on existing experiments or analysis, could the authors clarify which findings they expect to remain robust and which should be interpreted as specific to the current configuration?

1 comment

We expect the main qualitative finding is robust: text-to-image diffusion is compute-optimal at a much higher TPP than LLMs. The specific value of ~200 TPP may drift slightly depending on the chosen configurations, just like for LLMs (Hoffmann et al., 2022; Porian et al., 2024). We stress that the discussion below reflects our intuitions and one-off experiments and not the results of controlled ablation.

Architecture. We expect that reasonable changes to the architecture (±20% width/depth, normalization, DiT vs. MMDiT) will not substantially change our conclusions, in analogy to findings in the LLM literature (Kaplan et al., 2020). MoE is a sufficient departure from dense architectures that it needs its own scaling analysis, and we expect MoE to improve FLOP efficiency like it does in LLMs (Abnar et al., 2025; Tian et al., 2025).

Timestep schedule. The timestep schedule likely matters quite a bit. We proxy-tuned the timestep schedule in this paper, and we expect the optimal schedule to depend on the data, not the model size. But because the timestep schedule determines the shape of the loss response curve, we expect that changes to the timestep schedule will have some impact on the optimal TPP, although we are not sure how much. There isn’t a clear analog of the timestep schedule in the LLM literature.

Tokenizer, text encoder, and caption quality. We believe these three axes will affect the fitted laws, but we are not sure how significantly. In our opinion these three axes are the most interesting directions for follow-up work. In the LLM literature the tokenization and text-encoder are one and the same, and they can affect performance and likely need to be jointly scaled (Tao et al., 2024). The only analog to caption quality for an LLM is the data quality itself. In general text quality affects the scaling law by shifting the intercept of the scaling curves (Hestness et al., 2017; Subramanyam et al., 2025). It is an open question in T2I what the role of the captions is in determining the optimal TPP.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion

  • Two reviewers (3SAA and egsC) asked how robust we expect our 200 TPP finding to be to changes in training settings and architecture. We expect the qualitative findings to be robust and we include details in the discussions below.
  • Reviewer A6Ta requested additional analysis on the loss penalty data. We ran the analysis and even under the least favorable reading of the data our conclusions still hold (see our discussion with A6Ta for more details).
  • Two reviewers (egsC and A6Ta) noted that the wording regarding extrapolating our results to a hypothetical joint understanding / generation crossover-points in Sections 4.4 and Appendix D could be read as a statement of fact. This was not our intent as this point is merely conjectured to exist as a consequence of extrapolating our fits. We have improved the language in these sections to make this more clear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.