- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper finds an interesting insight that good performance in class-to-image ImageNet doesn't necessarily mean great performance in a text-to-image setup too. As such, the paper proposes to benchmark both class-conditional metrics and text-conditional metrics when new methods are proposed. The paper also retrained and evaluated many popular methods using a common setup to provide ease of use for future methods.
Critical strengths
- The paper showcases an interesting insight that there is a re-ranking in models between class-conditional benchmarks and text-conditional benchmarks. As such, the paper proposes to evaluate any new model on both setups to see if the proposed idea is sound or not.
- I really appreciate the authors’ retraining most of the popular methods on a common setup and evaluating them all, making it easier for future researchers.
Critical weaknesses
- It is really nice to see effort towards evaluating class-conditional method ideas on text-to-image scenarios too. However, given that the core motivation of the paper is about finding a text-to-image benchmark, I was wondering why ImageNet itself (with recaptioning) was not utilized as the dataset for this, similar to [1]. This would have been a clear comparison, as then, apart from all the t2i metrics, the class-conditional ImageNet metrics could have also been adopted to evaluate the t2i models. It would be really nice if the authors could comment on this. Additionally, it would also be nice to know the thoughts of the authors regarding the GPIC dataset[2]; which is now an open-source alternative to prior text-to-image datasets, and why the paper didn't use GPIC instead. Yes, I agree that the class-conditional setup could have been a bit problematic; but reusing a captioner to make K different classes could have been a solution too.
- I like the core idea of the paper of fixing most things to make everything as reproducible as possible. However, I am not so certain about fixing the CFG scale to 6.0 for all the t2i models. I fear that different models would have different optimal CFG, and this is not at all an ideal solution to fix it. This is why I feel that one of the major things missing from the paper is a w/o CFG comparison on both axes when comparing ImageNet FID and t2i metrics. I understand that for practical t2i purposes, CFG makes a lot of sense, but I feel this would actually show the benefits of each training method. It would be super nice if the authors could comment if they did some evaluations with both setup without CFG and if there was still the aspect of re-ranking! This is gonna be super useful too, as methods like RAE, which struggle with guidance, can still show proper improvements. I totally understand the restrictions of the rebuttal, so only a few evaluations and text explanations are more than enough!
- I really appreciate the authors’ approach of unifying everything into one style of model (DDT + in-context cond). However, I am just skeptical that methods like REG, which have the extra DINO class tokens, might get affected by it. Did the authors try updating REG to have K times the DINO class tokens such that it ain’t lost amongst the normal class tokens? From Table 2, it seems that the baseline is very close to REG with SD-VAE-EMA, which is not something I would have expected! A comment on this is really important for the analysis proposed in the paper.
- I like the benchmark. However, I am curious if it would make more sense to have more diverse metrics, including some aesthetic metrics like HPSv3 in the evaluation too. I would be super curious to know if FID improvement improves in aesthetic metrics but not in alignment metrics (I agree that HPSv3 does also have some sort of alignment, but it is still much more quality-based than GenEval). Additionally, I was also wondering if it makes sense to have a long caption evaluation and a short caption evaluation of the text-to-image metrics and having an average score for both as the final benchmark score. It would be super cool if the authors could comment on this. Again, similar to before, simple experiments with text-only explanations suffice!
References
[1] Degeorge, Lucas, et al. "How far can we go with ImageNet for Text-to-Image generation?." arXiv, 2025.
[2] Chandrasegaran, Keshigeyan, et al. "Gpic: A giant permissive image corpus for visual generation." arXiv, 2026.
Questions for authors
The idea of a common benchmark is interesting. Even though most recent papers tend to evaluate their ideas using both class-conditional and text-conditional setups, there have been many prior methods that only focused on FID boosting in the class-conditional setup. Unfortunately, in many cases we end up not knowing if they work in a text-conditional setup, and in many scenarios we also see significantly different setups to make them work. The proposed method’s goal of bringing everything under a single banner is pretty nice, and I really appreciate the effort. However, there are a few things I would like to know from the authors during the rebuttal.
(Note: I totally understand the restrictions of the rebuttal and simple text explanations and short experiments are more than enough)
- The authors used a combination of JourneyDB and BLIP-3o for their text-to-image setup. However, given the focus was mainly on benchmarking using open-source setups, I was curious why a recaptioned ImageNet dataset or GPIC was not adopted (Weakness 1) for training and evaluation. It would have also helped bring the evaluation metrics from class-conditional setups to the text-conditional setups. Additionally, I am a bit skeptical of the use of a fixed CFG for the text-to-image setting (Weakness 2) and wondered if it is possible to have the benchmark without guidance for proper evaluation.
- The authors used a single backbone based on DDT along with in-context conditioning for their benchmark. Even though it's extremely nice, I was wondering if this resulted in some benchmarks underperforming, like REG, which has an extra dino class token to denoise (Weakness 3). Additionally, I am also curious if it makes more sense to include aesthetic metrics in the benchmark (Weakness 3). I am hoping for comments from the authors regarding these in the rebuttal.
Even though I liked the ideas of the paper, because of the prior points I would recommend Weak Rejection. However, I am very interested in learning more from the authors in their rebuttal.