acceptodds
Under review as a conference paper at ICLR 2027

GenOlympiad: Benchmarking Frontier Image Models at Their Capability Boundaries

Abstract

Text-to-image models have made tremendous progress and can produce high-quality images. However, a question which would be asked in any other discipline is where the frontier models fall short. We introduce GenOlympiad, a benchmark which provides the most challenging text prompts from nine capability axes for stress-testing frontier text-to-image models. The assessed capabilities include generating large quantities, performing deep reasoning, ensuring hard physical consistency, following complex rules and so on. The prompts are designed such that prompt alignment is visually verifiable (e.g., counting) instead of relying on subjectiveness (e.g., style). Based on this, we develop an automated evaluation framework which decomposes the prompt into easy and hard questions, and for most hard questions, draws help lines to ground evidence for accurate grading. We evaluate 20+ strong text-to-image models, including Nano Banana Pro, GPT-Image-2.5, Seedream-5.0 and so on. We report that even the best-performing T2I model (Nano Banana Pro) only achieves a strict score of 23.9 out of 100 and that coding models such as GPT-6 Astra have higher prompt alignment scores (but not as visually pleasing). We also derive insights in model evolution, alignment with Elo scores, and effectiveness of our grounding-based evaluation protocol.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Image Generation, Benchmark, Automatic Evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_GLXaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GenOlympiad, a benchmark intended to expose capability-boundary failures of frontier text-to-image systems across nine axes. It proposes an automatic evaluating pipeline that decomposes prompts into easy gating questions and grounded hard-question checks followed by deterministic scoring.

Critical strengths

The grounded evaluation design is a substantive step beyond asking a VLM to directly judge complicated constraints. I agree that the next step in AIGC image evaluation is to move from image-level to region-level evaluation.

The benchmark also has a clear and consequential empirical message. Even the strongest evaluated text-to-image model achieves a very low score, with markedly different failure profiles across the nine axes. This makes the benchmark more diagnostic. The authors also evaluate a broad range of frontier models.

The results suggest that using coding tools to generate visual drafts can be a useful approach to image generation, particularly when prompt alignment is important.

Critical weaknesses

Thank you for the detailed rebuttal. The additional experiments on withheld tasks and evaluation variance have satisfactorily addressed my concerns regarding ranking robustness and evaluator reproducibility.

However, my main concern regarding the motivation and practical relevance of the benchmark remains. Given the remarkable progress of current text-to-image models, I believe the community is increasingly interested in their ability to satisfy real-world production requirements, rather than simply solving increasingly difficult synthetic tasks. While AI-assisted brainstorming can generate diverse and challenging prompts, it does not necessarily capture what users and professionals actually need in production scenarios. The examples provided by the authors are plausible, but I would have liked to see more systematic evidence from real users or practitioners to justify the selection of these nine capability axes.

Overall, I appreciate the improvements made in the rebuttal, particularly regarding evaluation reliability. Nevertheless, I remain concerned that the benchmark's practical motivation and construct validity are not sufficiently established. I therefore maintain my original rating of 2.

Questions for authors

No further questions.

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for the detailed questions.

W1 / Q1. Do the tasks reflect capabilities that users care about? What are the practical benefits?

We think the benchmark is useful in practice, because its scores agree with what human users prefer. We compare GenOlympiad scores with three human-based arenas. In these arenas, users compare two images and vote for the better one. The table below shows the Spearman correlation from Fig. 8.

BenchmarkArtificial Analysis ArenaLMArenaDesign Arena
GenOlympiad0.890.900.85
OneIG-Bench0.850.820.53

In other words, a model which does well on our hard prompts is also a model which humans prefer.

This ability also matters to real users. GenAI-Bench was built together with professional designers, and it finds that user prompts often need counting, comparison and logic [1].

The tasks are designed to reflect real uses. When the agent brainstorms a task, it must start from a real scene. The scenes cover many fields and industry, such as sport, medical use, comics, puzzles and infographics. The agent must also say in one sentence who would need this picture and what they would do with it (Appendix A.4). Here are some examples from our tasks.

  • A syringe must have a scale which can be read as an exact value.
  • A results board must match the results of the games, following the given game rules.
  • A restaurant menu must mark the right allergens for each dish.

For the practical benefits, the benchmark tells developers where to improve. For example, when GPT-5.6 Sol rewrites the prompt first, the strict score of GPT-Image-2.5 increases from 14.9 to 33.0. So reasoning and planning is a main weakness of current T2I models.

W2 / Q2. The motivation is weak. The prompts come from brainstorming with Claude and from filtering with five models.

Our motivation is that easy prompts can no longer distinguish between frontier models. For example, recent models already get about 0.90 on GenEval. But the best T2I model only gets 23.9 out of 100 on GenOlympiad. This gap shows that a large part of T2I ability is still not measured.

The brainstorming with Claude is indeed a part of our method. We started from many samples where frontier models fail, and wrote example tasks by hand (Appendix A.4). An agent then wrote more candidates from them. Finally, we used the scores of frontier models to remove the easy ones. This kind of filtering is also used by other benchmarks. For example, Humanity's Last Exam tests each question against frontier models, and rejects the questions which the models can answer [2].

The tasks are not only hard for the five filtering models. For example, Nano Banana Pro is not used for filtering, and its strict score is only 23.9. All the open-weight models are below 4 in strict score.

References

[1] Li et al. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. arXiv:2406.13743, 2024.

[2] Phan et al. A benchmark of expert-level academic questions to assess AI capabilities (Humanity's Last Exam). Nature, 2026.

AuthorsOpenReview ↗

Author Response (2/2)

W3. Different models are evaluated on different subsets. Is the comparison fair?

The ranking almost does not change if all models are scored on all 801 tasks. We recomputed Table 1 by making every withheld task counted as 0. We show ten of the rows below.

ModelStrict (Table 1)Strict (all as 0)Relaxed (Table 1)Relaxed (all as 0)
GPT-6 Astra40.536.356.150.4
GPT-5.6 Sol37.535.152.449.1
GPT-5.6 Sol rewrite33.030.848.945.7
Nano Banana Pro23.922.540.037.6
GPT-Image-2.514.913.929.627.4
Seedream 5.0 Pro15.114.028.426.4
Nano Banana 212.411.125.422.8
GPT-Image-210.29.523.421.7
FLUX.2 dev3.53.010.38.8
Sana 1.6B1.41.03.62.6

Over all 25 rows, the Spearman correlation with the ranking in Table 1 is 0.993 for strict score and 0.992 for relaxed score. Only a few models with very close scores change their places. We will add the full table to the appendix.

W4 / Q3. If we run the evaluation five times, do we get the same scores?

The scores are slightly different during the five runs. The standard deviation is between 0.5 and 1.4 points. We got this number by the following experiment. We use Gemini 2.5 Flash with temperature 0, which is the setting of Table 1. We took the same generated images and ran the full evaluation five times. Each run starts from scratch. We did this for four T2I models on a fixed random subset of 200 tasks.

Relaxed score on the 200 tasks.

ModelRun 1Run 2Run 3Run 4Run 5MeanStd
Nano Banana Pro38.239.439.338.840.239.20.7
GPT-Image-2.531.930.131.828.529.830.41.4
Seedream 5.0 Pro30.730.029.430.330.230.10.5
FLUX.2 dev11.710.711.911.410.611.30.6

Strict score on the 200 tasks.

ModelRun 1Run 2Run 3Run 4Run 5MeanStd
Nano Banana Pro23.124.322.422.722.423.00.8
GPT-Image-2.518.315.917.115.414.916.31.4
Seedream 5.0 Pro16.115.717.415.817.016.40.7
FLUX.2 dev4.33.63.23.22.63.40.6

Nano Banana Pro is the first in every run, and FLUX.2 dev is the last in every run. GPT-Image-2.5 and Seedream 5.0 Pro have almost the same score, so their order changes between runs.

We understand the concern about using a closed-source VLM. So we will release the whole evaluation programs, all generated images, and the VLM answer to every question. With these records, everyone can recompute our scores without calling the API.

Reviewer_AAmGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a benchmark containing 801 prompts for evaluating frontier models. It conducts detailed evaluations of both closed-source and open-source models, validates the alignment between evaluation results and human judgment, and also evaluates the current Prompt Engineering (PE) module and coding models. From the perspective of benchmark completeness, it is adequate. However, from the perspective of measuring contribution to the community, it is somewhat weak. The core of the paper’s solution is to use extensive grounding to verify whether some defined hard cases are satisfied, such as whether the number of dice and the corresponding pips meet expectations. This capability is not fundamental to whether a generative model generates well. It can admittedly evaluate certain capabilities, but for frontier models in user usage scenarios, why would such cases be considered? That is, the authors can evaluate this capability, but they should consider more what directions users care about in practical use. This is the meaning of a benchmark. As the authors say in the limitations, evaluating a case requires many VLM calls, which consumes resources. For example, generating 23 dice and 230 dice differ in difficulty, so under the current pipeline the number of VLM calls required would be significantly larger. From this perspective, this type of evaluation is meaningless.

Critical strengths

Under the authors’ setting, the experiments are very thorough, and many models are considered. The authors appropriately follow technological developments, for example by considering the impact of PE and coding models.

Critical weaknesses
  1. The proposed benchmark lacks thinking from the user perspective, and is even somewhat pedantic. We certainly admit that aesthetics and visual quality are subjective, but is a model that can perfectly satisfy the 23-dice case really becoming stronger and stronger?
  2. Because of the constraints in the prompts, a large part of the evaluation depends on grounding. The authors’ Table 2 ablation also shows that without grounding the overall effect would be much worse, and the authors mention the need for frequent VLM calls. This raises serious concerns about evaluation efficiency and resource consumption.
Questions for authors
  1. Do the authors believe that future progress of frontier models can be directly assessed through benchmark scores? Why?
  2. Could the authors elaborate in detail on why these concerns constitute the 801 prompts? The authors spent a great deal of effort validating the question-answering process, but did not validate whether the questions themselves are key.
1 comment

Thank you for the detailed questions.

W1. The user perspective and the benchmark is even somewhat pedantic.

We think the benchmark is useful in practice, because its scores agree with what human users prefer. We compare GenOlympiad scores with three human-based arenas. In these arenas, users compare two images and vote for the better one. The table below shows the Spearman correlation of Fig. 8.

BenchmarkArtificial Analysis ArenaLMArenaDesign Arena
GenOlympiad0.890.900.85
OneIG-Bench0.850.820.53

In other words, a model which does well on our hard prompts is also a model which humans prefer.

This ability also matters to real users. GenAI-Bench was built together with professional designers, and it finds that user prompts often need counting, comparison and logic [1].

A good benchmark needs tasks which are hard and can be checked. SWE-bench follows the same idea, where tasks should be hard for existing models but easy to verify [2]. GenOlympiad follows this principle for T2I models.

The tasks is design to be reflecting real uses. When the agent brainstorms a task, it must start from a real scene. The scenes cover many fields or industry, such as sport, medical use, comic for fun, puzzles, or infographics. And the agent itself must also say in one sentence who would need this picture what they would do with it (Appendix A.4).

The 23 dice example is one kind of task of puzzles. And only 32 of the 801 tasks are about counting objects. We understand that these counting tasks may look a bit pedantic. So we also checked the results without them. If we remove them, the ranking of models is almost the same. Most tasks are not about counting. Here are some examples from our tasks.

A syringe must have a scale which can be read into a exact value.

A results board must match the results of the game, following the specific game rules.

A restaurant menu must mark the right allergens for each dish.

W2. Evaluation efficiency.

The cost of our benchmark is indeed more expensive than other benchmark. But the cost is still controllable since a model only need to be evaluated once. Evaluating one model on all 801 tasks costs about 40 US dollars. It takes about 3 hours with about 100 parallel calls, and it needs no human.

The VLM reads are needed for accuracy. On the questions which need evidence grounding, the agreement with humans is 68.4% if we directly ask the VLM. With evidence grounding, it is 91.5%, as shown in Table 2.

Q1. Can the progress of frontier models be measured by the benchmark score?

Yes, we believe so. Newer models generally have higher scores. For example, the strict score increases from 5.2 for GPT-Image-1 to 14.9 for GPT-Image-2.5. In addition, the best T2I model only gets 23.9 out of 100, so there is still much room for future models.

Q2. Why these 801 prompts?

We choose these prompts because easy prompts can no longer distinguish between frontier models. For example, recent models already get about 0.90 on GenEval. So we started from cases where frontier models fail, and wrote example tasks by hand (Appendix A.4). An agent then wrote more candidates from them. Finally, we used the scores of frontier models to remove the easy ones. Now the best T2I model only gets 23.9 out of 100. This gap shows that a large part of T2I ability is still not measured.

You also ask whether the questions themselves are key. We checked the correlation between our benchmark with human-based arena as shown in table in W1. And we have a high correlation between AA Arena, LM Arena and Design Arena. Thus we believe that the questions are key.

References

[1] Li et al. GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation. arXiv:2406.13743, 2024.

[2] Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.

Reviewer_zXGiOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper provides a new, challenging benchmark for evaluating t2i models. It focuses on evaluating t2i models along axes which are more objective (e.g. counting objects in a scene) rather than subjective (e.g. style). It also involves a VLM-backed automated eval framework.

Critical strengths

Objective evaluation frameworks for frontier models are very useful to the community, which is why I'm recommending that this paper be accepted. this paper picks a well-defined scope (tasks which are verifiable, such as counting objects in the scene) which makes the paper clear and easy to understand as well. I also appreciate how the authors confirmed, using human raters, how the VLM performed (>90% agreement between the human raters and the VLMs). I also really like Figure 1, I think it does a great job illustrating the range of the GenOlympiad benchmark.

Critical weaknesses

Comparing the coding models to the t2i models (table 1) feels a bit unfair, as it's not too surprising they will do better in this verifiable domain setting (at the cost of aesthetics). that being said, I think it's fine to keep it, it just might be worth highlighting more in the paper why those models are expected to do so much better, as well as the trade-offs.

Questions for authors

Do you think there is some way to extend this framework to non-verifiable domains (such as aesthetics)? To be clear, I don't think this needs to go in the paper, I'm just curious to hear the team's thoughts.

1 comment

Thank you for the positive review.

W1. Comparing coding models with t2i models feels a bit unfair.

You are right. Coding models have a clear advantage in our setting, because they can compute the accurate coordinates, counts and layouts with code. A t2i model cannot do this explicitly. We did not mean to say that coding models are better than image models. We include them as a reference, to show how far a system can go when the reasoning is done by code. We will talk about this more clearly in the paper.

Q1. Can the framework be extended to non-verifiable domains such as aesthetics?

Thank you for this question. We think our framework can do some help in these domains.

One possible way is to check some simple rules of these domains. Our evaluation can find the objects, the lines and the text in an image. It can also measure their positions and colors. So perhaps we can manually design several rules to decide which pattern of the picture is good or bad.

Take aesthetics as an example. A simple rule of design is the golden ratio. We first find the box of the main object. Then we check whether this object is placed at 0.618 of the image width.

However, rules is hard to design. Taking symmetry as an example, a symmetric layout often looks good. But a face which is perfectly symmetric often looks strange, and a small asymmetry makes it look more natural. So more symmetry is not always better, and less symmetry is not always better either. Such a relation is hard to write as a number. This is why we think it could be very hard to find enough good rules.

So the simple rules only cover a small part of these domains. For the rest, we think human votes are still needed.

Reviewer_JdZLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GenOlympiad, a new benchmark with 801 challenging prompts across nine capability axes to test frontier text to image models. Since existing benchmarks like GenEval are almost saturated, the authors focus on hard constraints that can be visually verified, such as exact counting, latent computation, and physical consistency. To evaluate the generated images reliably, they build an automated pipeline that splits each task into easy gating questions and hard questions. Instead of just asking a VLM directly, the pipeline draws visual help lines or bounding boxes on the image, uses Gemini 2.5 Flash iteratively to verify the grounding, and then runs a deterministic Python script to calculate the strict and relaxed scores. They evaluate over 20 T2I models, two coding models, and prompt engineering methods, showing that even the best T2I model Nano Banana Pro only gets 23.9 strict score. They also validate their evaluator against human judgements on Prolific.

Critical strengths

First, the evaluation pipeline that seperates perception from calculation is very well designed. VLM judges are usually bad at counting or spatial math on their own. By letting the VLM only verify bounding boxes or helper lines iteratively and leaving the actual math or logic to a deterministic Python script, the evaluation becomes much more reliable. Table 2 clearly shows that this grounding step improves human agreement on hard questions from 68.4 percent to 91.5 percent, which makes the benchmark results convincing.

Second, the benchmark is genuinely challenging and the experiments give nice insights into where current models fail. Testing coding models like GPT 6 Astra and prompt rewriting with GPT 5.6 Sol on GPT Image 2.5 is a smart idea. It clearly shows that a lot of the failures in current T2I models comes from weak reasoning and planning rather than just image rendering, since rewriting the prompt alone boosts GPT Image 2.5 strict score from 14.9 to 33.0.

Critical weaknesses

First, the way withheld tasks are handled makes the comparison across models a bit unfair. When grounding fails or when a prompt is too long for an open weight model context length, the task is simply removed from the overall score calculation instead of getting a zero. In Figure 6, strong models only have around 5 percent withheld tasks, while weaker models have up to 27.5 percent removed. This means different models are scored on different subsets of the benchmark. Also, as mentioned in Section 3.3, grounding often fails when generated objects are too small, overlapping, or unrecognizable. Those cases sounds like generation failures of the T2I model that should be penalized with a 0 rather than excluded.

Second, there is a potential selection bias in how the dataset was filtered, and the evaluation cost is quite high. In Appendix A.3, the authors filter the difficulty of the prompts using five specific models including GPT Image 2 and Nano Banana 2. Even though images were regenerated later, picking tasks specifically where these five models fail might unfairly lower their scores compared to models not used in the filtering step like Nano Banana Pro. Also, needing 59.5 VLM reads per image on average means almost 48000 API calls just to test one model on 801 prompts, which is pretty heavy for practical usage.

Questions for authors

First, for clarification, what happens to the overall strict and relaxed scores in Table 1 if all withheld tasks from both grounding failures and context length limits are counted as 0 instead of being dropped, and does this change the model rankings?

Second, in Section 3.3 you mention grounding fails when objects overlap too much, have bad contrast with the background, or do not match the expected objects. Why are these cases not treated as a 0 score for the T2I model, or caught earlier by the easy gating questions?

Third, since you used five specific models including Nano Banana 2 and GPT Image 2 to filter out easy tasks in Appendix A.3, how much selection bias does this create against them compared to models not in that filter like Nano Banana Pro and GPT Image 2.5?

Fourth, can you explain why the pipeline needs 59.5 VLM reads per image on average when there are only 2.2 hard questions per prompt? Is it checking every single bounding box in a separate query, and what is the maximum retry limit for the iterative grounding? For future work, it might be helpful to batch object crops together to reduce the number of VLM calls.

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for the detailed questions.

W1 / Q1. What happens if all withheld tasks are counted as 0?

The scores become lower, but the ranking almost does not change. We recomputed Table 1 by making every withheld task counted as 0. We show ten of the rows below.

ModelStrict (Table 1)Strict (all as 0)Relaxed (Table 1)Relaxed (all as 0)
GPT-6 Astra40.536.356.150.4
GPT-5.6 Sol37.535.152.449.1
GPT-5.6 Sol rewrite33.030.848.945.7
Nano Banana Pro23.922.540.037.6
GPT-Image-2.514.913.929.627.4
Seedream 5.0 Pro15.114.028.426.4
Nano Banana 212.411.125.422.8
GPT-Image-210.29.523.421.7
FLUX.2 dev3.53.010.38.8
Sana 1.6B1.41.03.62.6

Over all 25 rows, the Spearman correlation with the ranking in Table 1 is 0.993 for strict score and 0.992 for relaxed score. Only a few models with very close scores change their places. We will add the full table to the appendix.

Q2. Why are these cases not scored as 0, and why are they not caught by the easy questions?

The easy questions and the grounding look at the image in different ways. An easy question looks at the whole image and asks a simple thing, for example whether there are many bottle caps on a table. So an image can pass the easy questions but still have a problem on hard question. These problem will then affect grounding. For example, some caps are covered by other caps. The program tries several times, but these caps still cannot be boxed one by one. In the end, the grounding fails.

In more than half of the withheld tasks, the easy questions have passed but the grounding fails during hard questions. At this point the program cannot tell which side is wrong. Maybe the T2I model may have drawn the objects badly. For example, two bottle caps may appear fused together, making it difficult to determine whether they should be counted as one object or two. The object may also be correct but just too small, or it is correct but part of the object covered by another object. So we do not give a score of 0 here.

W2 / Q3. How much selection bias is there against the five filtering models?

We agree that the filtering lowers the scores of the five models. And we estimated that the selection bias may roughly drop 3 to 4 points relaxed score.

We got this number by the following method. Since the five models are already filtered, so the analysis on these five model may not be reliable. Thus, we took five frontier models which are not used in our filtering. They are Nano Banana Pro, GPT-Image-2.5, Seedream 5.0 Pro, Nano Banana and GPT-Image-1. We used them to filter the 801 tasks in a similar way. Then we measured how much the relaxed score of a model drops when this model is used for filtering. If only one model is used for filtering, its score drops by about 20 points. If four or five models are used, the score drops by about 3 to 4 points. So the selection bias against the five filtering models is dropping the score by 3 to 4.

AuthorsOpenReview ↗

Author Response (2/2)

Q4. Why does the evaluation need 59.5 VLM reads per image? What is the maximum retries?

Your guess is right, and each box is checked in a separate query. One question is often about many objects. On average there are 2.2 hard questions per prompt, but each question may need many reads. For one question, the evaluation program has three steps.

  1. The program draws the helping lines or boxes on the image.
  2. The VLM checks whether they are correct. If the VLM does not accept them, the program draws the helping lines again.
  3. The program crops each confirmed box and reads it in a separate query.

The maximum number of trials is 3 times. If the VLM still does not accept the boxes after the last trial, we withhold the task.

We also tested whether more trials help. For each of the 25 models in Table 1, we ran the grounding again on its withheld tasks.

The table below shows the average over the 25 models, we can further try 6 as the maximum number of trials to see what happen on the withhold tasks.

Trial123456
Withheld tasks, average of 25 models (% of the 801 tasks)23.017.915.714.413.713.4

Each new trail helps less. The second trail reduces the withheld tasks from 23.0% to 17.9%. The sixth trail only reduces them from 13.7% to 13.4%. So too much trials bring very little improve.

Evaluating one model on 801 tasks costs about 40 US dollars and takes about 3 hours with about 100 parallel calls. Besides, Thank you for the suggestion of batching several crops into one query. We think it is a good way to reduce the cost in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.