- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PixelBench evaluates language-model agents across 75 pixel-art generation tasks and 25 editing tasks. Models write Python code using a shared drawing API, with strict constraints enforced on canvas size and color palette. Generation follows a cycle of inspecting the canvas, editing code, and re-rendering outputs. Quality assessment relies on full human ranking of 1200 samples from four annotators, alongside a multimodal judge calibrated to match this human consensus. Twelve different agent setups are benchmarked, and GPT-6 Astra leads both the automated judge and human rankings. The tasks, generated art, ranking data, and judge evaluation protocol are all publicly released.
Critical strengths
Every model runs inside the identical environment. Any output with the wrong canvas size or excess colors gets rejected at the finish step, so the leaderboard stays clean and avoids entries that just output smooth PNGs and pretend they are pixel art. Since everything comes from executable code, results are easy to inspect and rerun.
Their judge calibrates against full ranked lists instead of a small set of pairwise preferences. The analysis tracing how rendering margins, scoring rubrics, and reference examples shift judge–human agreement is the piece I would most likely reuse elsewhere.
Critical weaknesses
The claims come across as more general than the experimental setup supports. What they measure is how well an agent can operate this specific drawing library under fixed compute limits, not pixel-art capability broadly. I’d like to see at least one stress test, such as swapping between a more capable and a stripped-down API, or doubling the edit budget. This matters especially for open-weight agents, which often take longer to converge.
The reference images shown in the ranking interface introduce a confound. For each generation task, raters see a GPT-Image-2 rendering of the same prompt, which can push them to rank outputs based on how closely they match that particular look. A blind reranking of a subset without these references would reveal how much the human ordering shifts.
Questions for authors
We ask for three additional analyses. First, across 20 generation tasks, quantify how much the average human rank shifts once the GPT-Image-2 reference is removed from the annotator interface. Second, split Kendall’s τ and judge‑human agreement by subject category and canvas size, separating 16–32 px canvases from those 64 px and above. The manuscript already suggests stronger agreement for larger, concrete subjects, so we would like supporting numerical breakdowns. Third, under a doubled code write-and-edit budget, check whether mid-tier open-weight models narrow the gap against GPT-6 Astra or if the performance difference remains roughly unchanged.
6 comments
(2/4) W1 + Q3 (continued)
2. More reasoning. We ran three models at further reasoning-effort levels on all 100 tasks: GPT-6 Astra at low, medium, high, and xhigh; DeepSeek V4 Flash with reasoning switched off and at low, high, and max; and GLM-5.3 Flash at low, high, and max. GLM-5.3 Flash rises at every level (62.6, 69.8, 75.0), yet at max it still trails GPT-6 Astra by 8.8 points; DeepSeek V4 Flash saturates at high, with max adding only 0.6. Conversely, GPT-6 Astra at its lowest level, low (81.8), is still above every other configuration. Figure 10b of the revision shows each model's levels side by side on six randomly drawn tasks, and we invite the reviewer to take a look.
3. A poorer API: direct answering. As the poorest possible interface, we designed a direct-answer protocol. In one response, the model declares a palette with one character per color and writes every pixel row at the native canvas size, frame by frame for multi-frame tasks, with the original of an editing task given in the same format. There are no tool calls, no code execution, no view of the result, and no second turn. The answer is parsed into a pixel image and scored by the same judge, and a malformed answer, or one that exceeds the token limit, counts as a failure.
| Configuration | Tasks | Direct answer | Agent |
|---|---|---|---|
| DeepSeek V4 Flash | Generation | 17.1 | 66.9 |
| DeepSeek V4 Flash | Editing | 23.0 | 61.4 |
| GPT-6 Astra | Generation, 32×32 and below | 80.8 | 81.4 |
| GPT-6 Astra | Generation, above 64 pixels | 69.1 | 83.1 |
| GPT-6 Astra | Editing | 78.1 | 88.0 |
With the poorer interface the two models keep their order and the gap widens sharply, from 83.8 against 65.5 overall to 76.9 against 18.6. The weaker model can hardly complete the tasks through this interface, and the stronger one drops clearly on large canvases. Appendix J.2 (Table 10) of the revision gives the results by canvas size.
(3/4) W2 + Q1. Are the reference images in the ranking interface a confound?
Reviewer's concern. For generation tasks, the ranking interface shows a GPT-Image-2 rendering of the same brief, so the raters may end up ranking proximity to that look. A no-reference rerank on a subset would show how much the human order moves.
Reviewer's question. "On 20 generation tasks, how much does mean human rank change without the GPT-Image-2 reference?"
Response. Thank you for this suggestion, which we built into the new human rating. Without the reference, the human order barely moves: on the 75 generation tasks, a configuration's mean rank changes by 0.35 places on average, and the leaderboards of the raters with and without references correlate at Spearman ρ = 0.965, with the same first place.
We ran a fuller version than requested: two of the ten raters ranked all 75 generation tasks, and all 25 editing tasks, without any reference. The numbers above compare the twelve configurations' mean ranks from the eight raters with references and from the two without.
For comparison, there are 45 ways to split the ten raters into two and eight. The other 44 splits give generation leaderboards that correlate at 0.930–0.993, and the split by reference falls inside this range.
Appendix I of the revision reports this comparison, together with the judge's agreement with each group of raters.
Response to Reviewer ZG59
We thank the reviewer for the precise and actionable suggestions. We respond to each point below.
(1/4) W1 + Q3. Are the claims broader than the setup, and would a doubled budget let mid-tier models catch up?
Reviewer's concern. The claims read broader than the setup: what is measured is the ability to drive this drawing library under this budget, which is some distance from pixel art in general. The reviewer asks for at least one stress test, such as a richer or poorer API or a doubled edit budget, especially for open-weight agents that may converge more slowly.
Response. We thank the reviewer for the concrete stress tests. We examined the gap from three sides, and in each it stays: even if every artwork that used up the write budget reached GPT-6 Astra's score on its task, no configuration would catch up; GLM-5.3 Flash, a mid-tier open-weight model, still trails GPT-6 Astra by 8.8 points at its highest reasoning effort; and with a poorer interface the gap widens. We also agree that the claims should match the setup: the revision scopes them to pixel-art creation under one programmatic protocol, states in Section 7 that there is a single environment, and leaves comparisons in other environments and agent frameworks to future work.
1. A larger write budget would not let the other configurations catch up. Even if every artwork that used up the write budget reached GPT-6 Astra's score on the same task, no configuration would catch up with GPT-6 Astra: none would gain more than 4.5 points, and GLM-5.3 Flash, a mid-tier open-weight model, would gain 3.8 points and still trail by about 5. The write budget is 25 writes for a generation task and 15 for an editing task. An artwork that stopped short of the budget was submitted by the model's own choice, so more writes could change only the artworks that used the budget up. GPT-6 Astra and GPT-5.6 Sol never do; such artworks come from the models that revise as they go. The table raises each of them to GPT-6 Astra's score on its task:
The behavior of the models also makes a change of conclusion unlikely. The artworks that used the full budget sit on tasks their models find hard: on the same task they trail GPT-6 Astra by 19.2 points on average, against 11.1 for the same models' other artworks. Comparing the canvas after every run of these 122 artworks, we find that the long processes do not converge steadily: on average 41% of the runs in the second half change nothing; 30% of the artworks return at some point to an earlier canvas; and 42% are still being redrawn after the midpoint, with more than half of their pixels different from the midpoint canvas, while only a third change fewer than 10% of their pixels after it. More writes would mainly prolong this back and forth.