- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GenOlympiad, a benchmark intended to expose capability-boundary failures of frontier text-to-image systems across nine axes. It proposes an automatic evaluating pipeline that decomposes prompts into easy gating questions and grounded hard-question checks followed by deterministic scoring.
Critical strengths
The grounded evaluation design is a substantive step beyond asking a VLM to directly judge complicated constraints. I agree that the next step in AIGC image evaluation is to move from image-level to region-level evaluation.
The benchmark also has a clear and consequential empirical message. Even the strongest evaluated text-to-image model achieves a very low score, with markedly different failure profiles across the nine axes. This makes the benchmark more diagnostic. The authors also evaluate a broad range of frontier models.
The results suggest that using coding tools to generate visual drafts can be a useful approach to image generation, particularly when prompt alignment is important.
Critical weaknesses
Thank you for the detailed rebuttal. The additional experiments on withheld tasks and evaluation variance have satisfactorily addressed my concerns regarding ranking robustness and evaluator reproducibility.
However, my main concern regarding the motivation and practical relevance of the benchmark remains. Given the remarkable progress of current text-to-image models, I believe the community is increasingly interested in their ability to satisfy real-world production requirements, rather than simply solving increasingly difficult synthetic tasks. While AI-assisted brainstorming can generate diverse and challenging prompts, it does not necessarily capture what users and professionals actually need in production scenarios. The examples provided by the authors are plausible, but I would have liked to see more systematic evidence from real users or practitioners to justify the selection of these nine capability axes.
Overall, I appreciate the improvements made in the rebuttal, particularly regarding evaluation reliability. Nevertheless, I remain concerned that the benchmark's practical motivation and construct validity are not sufficiently established. I therefore maintain my original rating of 2.
Questions for authors
No further questions.
2 comments
Author Response (2/2)
W3. Different models are evaluated on different subsets. Is the comparison fair?
The ranking almost does not change if all models are scored on all 801 tasks. We recomputed Table 1 by making every withheld task counted as 0. We show ten of the rows below.
| Model | Strict (Table 1) | Strict (all as 0) | Relaxed (Table 1) | Relaxed (all as 0) |
|---|---|---|---|---|
| GPT-6 Astra | 40.5 | 36.3 | 56.1 | 50.4 |
| GPT-5.6 Sol | 37.5 | 35.1 | 52.4 | 49.1 |
| GPT-5.6 Sol rewrite | 33.0 | 30.8 | 48.9 | 45.7 |
| Nano Banana Pro | 23.9 | 22.5 | 40.0 | 37.6 |
| GPT-Image-2.5 | 14.9 | 13.9 | 29.6 | 27.4 |
| Seedream 5.0 Pro | 15.1 | 14.0 | 28.4 | 26.4 |
| Nano Banana 2 | 12.4 | 11.1 | 25.4 | 22.8 |
| GPT-Image-2 | 10.2 | 9.5 | 23.4 | 21.7 |
| FLUX.2 dev | 3.5 | 3.0 | 10.3 | 8.8 |
| Sana 1.6B | 1.4 | 1.0 | 3.6 | 2.6 |
Over all 25 rows, the Spearman correlation with the ranking in Table 1 is 0.993 for strict score and 0.992 for relaxed score. Only a few models with very close scores change their places. We will add the full table to the appendix.
W4 / Q3. If we run the evaluation five times, do we get the same scores?
The scores are slightly different during the five runs. The standard deviation is between 0.5 and 1.4 points. We got this number by the following experiment. We use Gemini 2.5 Flash with temperature 0, which is the setting of Table 1. We took the same generated images and ran the full evaluation five times. Each run starts from scratch. We did this for four T2I models on a fixed random subset of 200 tasks.
Relaxed score on the 200 tasks.
| Model | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Mean | Std |
|---|---|---|---|---|---|---|---|
| Nano Banana Pro | 38.2 | 39.4 | 39.3 | 38.8 | 40.2 | 39.2 | 0.7 |
| GPT-Image-2.5 | 31.9 | 30.1 | 31.8 | 28.5 | 29.8 | 30.4 | 1.4 |
| Seedream 5.0 Pro | 30.7 | 30.0 | 29.4 | 30.3 | 30.2 | 30.1 | 0.5 |
| FLUX.2 dev | 11.7 | 10.7 | 11.9 | 11.4 | 10.6 | 11.3 | 0.6 |
Strict score on the 200 tasks.
| Model | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Mean | Std |
|---|---|---|---|---|---|---|---|
| Nano Banana Pro | 23.1 | 24.3 | 22.4 | 22.7 | 22.4 | 23.0 | 0.8 |
| GPT-Image-2.5 | 18.3 | 15.9 | 17.1 | 15.4 | 14.9 | 16.3 | 1.4 |
| Seedream 5.0 Pro | 16.1 | 15.7 | 17.4 | 15.8 | 17.0 | 16.4 | 0.7 |
| FLUX.2 dev | 4.3 | 3.6 | 3.2 | 3.2 | 2.6 | 3.4 | 0.6 |
Nano Banana Pro is the first in every run, and FLUX.2 dev is the last in every run. GPT-Image-2.5 and Seedream 5.0 Pro have almost the same score, so their order changes between runs.
We understand the concern about using a closed-source VLM. So we will release the whole evaluation programs, all generated images, and the VLM answer to every question. With these records, everyone can recompute our scores without calling the API.
Author Response (1/2)
Thank you for the detailed questions.
W1 / Q1. Do the tasks reflect capabilities that users care about? What are the practical benefits?
We think the benchmark is useful in practice, because its scores agree with what human users prefer. We compare GenOlympiad scores with three human-based arenas. In these arenas, users compare two images and vote for the better one. The table below shows the Spearman correlation from Fig. 8.
In other words, a model which does well on our hard prompts is also a model which humans prefer.
This ability also matters to real users. GenAI-Bench was built together with professional designers, and it finds that user prompts often need counting, comparison and logic [1].
The tasks are designed to reflect real uses. When the agent brainstorms a task, it must start from a real scene. The scenes cover many fields and industry, such as sport, medical use, comics, puzzles and infographics. The agent must also say in one sentence who would need this picture and what they would do with it (Appendix A.4). Here are some examples from our tasks.
For the practical benefits, the benchmark tells developers where to improve. For example, when GPT-5.6 Sol rewrites the prompt first, the strict score of GPT-Image-2.5 increases from 14.9 to 33.0. So reasoning and planning is a main weakness of current T2I models.
W2 / Q2. The motivation is weak. The prompts come from brainstorming with Claude and from filtering with five models.
Our motivation is that easy prompts can no longer distinguish between frontier models. For example, recent models already get about 0.90 on GenEval. But the best T2I model only gets 23.9 out of 100 on GenOlympiad. This gap shows that a large part of T2I ability is still not measured.
The brainstorming with Claude is indeed a part of our method. We started from many samples where frontier models fail, and wrote example tasks by hand (Appendix A.4). An agent then wrote more candidates from them. Finally, we used the scores of frontier models to remove the easy ones. This kind of filtering is also used by other benchmarks. For example, Humanity's Last Exam tests each question against frontier models, and rejects the questions which the models can answer [2].
The tasks are not only hard for the five filtering models. For example, Nano Banana Pro is not used for filtering, and its strict score is only 23.9. All the open-weight models are below 4 in strict score.
References
[1] Li et al. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. arXiv:2406.13743, 2024.
[2] Phan et al. A benchmark of expert-level academic questions to assess AI capabilities (Humanity's Last Exam). Nature, 2026.