acceptodds
Withdrawn from ICLR 2027

DiffusionBench: On Holistic Evaluation of Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: improvements on DiffusionBench are more likely to reflect broader progress.

Primary area:
datasets and benchmarks
Keywords:
diffusion transformers, class-conditional image generation, text-to-image generation, generative model evaluation, image generation benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1JycOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds an interesting insight that good performance in class-to-image ImageNet doesn't necessarily mean great performance in a text-to-image setup too. As such, the paper proposes to benchmark both class-conditional metrics and text-conditional metrics when new methods are proposed. The paper also retrained and evaluated many popular methods using a common setup to provide ease of use for future methods.

Critical strengths
  1. The paper showcases an interesting insight that there is a re-ranking in models between class-conditional benchmarks and text-conditional benchmarks. As such, the paper proposes to evaluate any new model on both setups to see if the proposed idea is sound or not.
  2. I really appreciate the authors’ retraining most of the popular methods on a common setup and evaluating them all, making it easier for future researchers.
Critical weaknesses
  1. It is really nice to see effort towards evaluating class-conditional method ideas on text-to-image scenarios too. However, given that the core motivation of the paper is about finding a text-to-image benchmark, I was wondering why ImageNet itself (with recaptioning) was not utilized as the dataset for this, similar to [1]. This would have been a clear comparison, as then, apart from all the t2i metrics, the class-conditional ImageNet metrics could have also been adopted to evaluate the t2i models. It would be really nice if the authors could comment on this. Additionally, it would also be nice to know the thoughts of the authors regarding the GPIC dataset[2]; which is now an open-source alternative to prior text-to-image datasets, and why the paper didn't use GPIC instead. Yes, I agree that the class-conditional setup could have been a bit problematic; but reusing a captioner to make K different classes could have been a solution too.
  2. I like the core idea of the paper of fixing most things to make everything as reproducible as possible. However, I am not so certain about fixing the CFG scale to 6.0 for all the t2i models. I fear that different models would have different optimal CFG, and this is not at all an ideal solution to fix it. This is why I feel that one of the major things missing from the paper is a w/o CFG comparison on both axes when comparing ImageNet FID and t2i metrics. I understand that for practical t2i purposes, CFG makes a lot of sense, but I feel this would actually show the benefits of each training method. It would be super nice if the authors could comment if they did some evaluations with both setup without CFG and if there was still the aspect of re-ranking! This is gonna be super useful too, as methods like RAE, which struggle with guidance, can still show proper improvements. I totally understand the restrictions of the rebuttal, so only a few evaluations and text explanations are more than enough!
  3. I really appreciate the authors’ approach of unifying everything into one style of model (DDT + in-context cond). However, I am just skeptical that methods like REG, which have the extra DINO class tokens, might get affected by it. Did the authors try updating REG to have K times the DINO class tokens such that it ain’t lost amongst the normal class tokens? From Table 2, it seems that the baseline is very close to REG with SD-VAE-EMA, which is not something I would have expected! A comment on this is really important for the analysis proposed in the paper.
  4. I like the benchmark. However, I am curious if it would make more sense to have more diverse metrics, including some aesthetic metrics like HPSv3 in the evaluation too. I would be super curious to know if FID improvement improves in aesthetic metrics but not in alignment metrics (I agree that HPSv3 does also have some sort of alignment, but it is still much more quality-based than GenEval). Additionally, I was also wondering if it makes sense to have a long caption evaluation and a short caption evaluation of the text-to-image metrics and having an average score for both as the final benchmark score. It would be super cool if the authors could comment on this. Again, similar to before, simple experiments with text-only explanations suffice!

References

[1] Degeorge, Lucas, et al. "How far can we go with ImageNet for Text-to-Image generation?." arXiv, 2025.

[2] Chandrasegaran, Keshigeyan, et al. "Gpic: A giant permissive image corpus for visual generation." arXiv, 2026.

Questions for authors

The idea of a common benchmark is interesting. Even though most recent papers tend to evaluate their ideas using both class-conditional and text-conditional setups, there have been many prior methods that only focused on FID boosting in the class-conditional setup. Unfortunately, in many cases we end up not knowing if they work in a text-conditional setup, and in many scenarios we also see significantly different setups to make them work. The proposed method’s goal of bringing everything under a single banner is pretty nice, and I really appreciate the effort. However, there are a few things I would like to know from the authors during the rebuttal.

(Note: I totally understand the restrictions of the rebuttal and simple text explanations and short experiments are more than enough)

  1. The authors used a combination of JourneyDB and BLIP-3o for their text-to-image setup. However, given the focus was mainly on benchmarking using open-source setups, I was curious why a recaptioned ImageNet dataset or GPIC was not adopted (Weakness 1) for training and evaluation. It would have also helped bring the evaluation metrics from class-conditional setups to the text-conditional setups. Additionally, I am a bit skeptical of the use of a fixed CFG for the text-to-image setting (Weakness 2) and wondered if it is possible to have the benchmark without guidance for proper evaluation.
  2. The authors used a single backbone based on DDT along with in-context conditioning for their benchmark. Even though it's extremely nice, I was wondering if this resulted in some benchmarks underperforming, like REG, which has an extra dino class token to denoise (Weakness 3). Additionally, I am also curious if it makes more sense to include aesthetic metrics in the benchmark (Weakness 3). I am hoping for comments from the authors regarding these in the rebuttal.

Even though I liked the ideas of the paper, because of the prior points I would recommend Weak Rejection. However, I am very interested in learning more from the authors in their rebuttal.

Reviewer_qRHhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper argues that the diffusion transformers field mainly relies on the class conditional image generation / imagenet FID. To solve this, the paper introduces the NanoGen, a unified training and evaluation codebase that switches from ImageNet to text-to-image and only the dataset and conditioner change. Along with this DiffusionBench combines existing benchmarks together and reports both ImageNet and text to image results.

Critical strengths
  • The over-reliance on the Imagenet is a biggest problem in the field as SoTA methods on imagenet not always translate at scale and it's difficult for academia to conduct experiments at scale. This paper attempts to tackle this crucial problem.
  • Extensive experiments are conducted against various methods and models, almost 21 in total.
  • The paper shows that T2I training is no more expensive than ImageNet under this recipe, which counters a common reason for skipping T2I.
  • The paper is well written and easy to follow.
Critical weaknesses
  • The findings are not particularly novel. Particularly, the correlation between FID vs. semantic benchmarks (geneval / dpgbench) is negatively & moderately correlated. ImageNet FID measures fidelity and diversity against a reference distribution. GenEval, DPG-Bench and GenAIBench measure prompt alignment and composition, not image quality.
  • The scope is narrower than the title suggests. Every method runs on one backbone (DDT with in-context conditioning) at 256 px with one text encoder. “Holistic evaluation of diffusion transformers” overstates this.
  • The experiments are on small scale and does not tell which methods does well when scaled.
  • Human evaluation study is missing. GenEval and DPG-Bench scores are typically not correlated with humans. And hence it is important to add Elo style ranking to provide holistic view.
Questions for authors
  • Can authors provide the CFG sweep for T2I?
  • What if we train longer? Will it affect the ranking & performance?
  • Can 256p resolution training will tell performance for higher resolution?
Reviewer_rnLvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether performance gains on ImageNet class-conditional generation transfer to text-to-image generation. The authors introduce NanoGen, a unified training and evaluation framework that supports various latent-space, pixel-space, and few-step generation methods, and combine both tasks into DiffusionBench. By retraining and comparing multiple models under a consistent setup, they examine the relationship between ImageNet FID and text-to-image metrics, including GenEval, DPG-Bench, and GenAIBench. Their results show that improvements on one task do not always translate to the other, highlighting the importance of evaluating DiT methods on both tasks to obtain a more comprehensive assessment of their effectiveness.

Critical strengths
  1. The paper addresses a meaningful question: whether improvements on ImageNet generalize to text-to-image generation.

  2. NanoGen unifies training and evaluation across both tasks, reducing the effort needed for cross-task comparisons.

  3. The experiments cover diverse generation methods and multiple metrics, providing useful insights into cross-task performance.

Critical weaknesses
  1. The main conclusion is stronger than the evidence supports. GenEval and DPG-Bench still show moderate correlations with ImageNet FID. Within the FID range of 1–2, the correlations are approximately −0.719 and −0.687, respectively, which contradicts the paper's claim that this region shows the weakest correlation.

  2. The CFG settings differ across tasks. The authors search for the optimal CFG for each method on ImageNet but use a fixed CFG of 6 for text-to-image generation. The guidance intervals also differ. Without a sensitivity analysis, it remains unclear whether these choices contribute to the observed differences in method rankings.

  3. Method rankings are sensitive to training budget. When training increases from 100K to 200K steps, FLUX.2-VAE's GenEval score drops from 0.675 to 0.625, while E2E-VAVAE improves from 0.632 to 0.679, reversing their rankings. The paper does not examine how such changes affect its main conclusions or report uncertainty estimates.

  4. Reproducibility details are incomplete. Key settings, including data mixing ratios, text-to-image warmup, latent normalization, and evaluator versions, are not clearly specified. These details are important for reproducing the results and adopting the proposed benchmark.

Questions for authors
  1. How do the authors reconcile the relatively strong correlations within the FID range of 1–2 with the claim that this region is the least correlated? Would the conclusions about ImageNet rankings still hold under Spearman or Kendall rank correlation analysis?

  2. Do the cross-task ranking differences persist after a limited CFG search for representative methods and alignment of the guidance intervals across tasks?

  3. Can the authors use the existing 100K/200K checkpoints to examine how rankings change with training budget and provide uncertainty estimates for key results?

  4. Can the authors provide the full training and evaluation configurations, particularly the data mixing ratios, text-to-image warmup, latent normalization procedure, and evaluator versions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.