acceptodds
Under review as a conference paper at ICLR 2027

PixelBench: Evaluating Language Models for Pixel-Art Creation

Abstract

Pixel art is widely used in games and digital content creation, requiring readable forms, coherent spatial relationships, and harmonious colors under limited resolution and palette constraints. Language models show growing promise in visual creation, yet their pixel art creation capabilities lack systematic evaluation. We introduce PixelBench, formalizing text-to-image generation and instruction-guided editing for pixel art and providing a benchmark for these tasks. We further present a creation environment in which language model agents invoke drawing tools through code to generate and edit pixel art. We assess the quality of generated artwork through human evaluation and employ a multimodal large language model as a judge, validating the agreement between its automated assessments and human preferences. We compare language models and configurations across diverse subjects and canvas sizes. In our experiments, GPT-6 Astra achieves both the highest average judge score and the best average human rank, highlighting the potential of general-purpose language models for pixel art creation. We release the tasks, all artworks, the human rankings, and the judge protocol.

Primary area:
datasets and benchmarks
Keywords:
language models, datasets and benchmarks, language model agents, pixel art

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ZG59OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PixelBench evaluates language-model agents across 75 pixel-art generation tasks and 25 editing tasks. Models write Python code using a shared drawing API, with strict constraints enforced on canvas size and color palette. Generation follows a cycle of inspecting the canvas, editing code, and re-rendering outputs. Quality assessment relies on full human ranking of 1200 samples from four annotators, alongside a multimodal judge calibrated to match this human consensus. Twelve different agent setups are benchmarked, and GPT-6 Astra leads both the automated judge and human rankings. The tasks, generated art, ranking data, and judge evaluation protocol are all publicly released.

Critical strengths

Every model runs inside the identical environment. Any output with the wrong canvas size or excess colors gets rejected at the finish step, so the leaderboard stays clean and avoids entries that just output smooth PNGs and pretend they are pixel art. Since everything comes from executable code, results are easy to inspect and rerun.

Their judge calibrates against full ranked lists instead of a small set of pairwise preferences. The analysis tracing how rendering margins, scoring rubrics, and reference examples shift judge–human agreement is the piece I would most likely reuse elsewhere.

Critical weaknesses

The claims come across as more general than the experimental setup supports. What they measure is how well an agent can operate this specific drawing library under fixed compute limits, not pixel-art capability broadly. I’d like to see at least one stress test, such as swapping between a more capable and a stripped-down API, or doubling the edit budget. This matters especially for open-weight agents, which often take longer to converge.

The reference images shown in the ranking interface introduce a confound. For each generation task, raters see a GPT-Image-2 rendering of the same prompt, which can push them to rank outputs based on how closely they match that particular look. A blind reranking of a subset without these references would reveal how much the human ordering shifts.

Questions for authors

We ask for three additional analyses. First, across 20 generation tasks, quantify how much the average human rank shifts once the GPT-Image-2 reference is removed from the annotator interface. Second, split Kendall’s τ and judge‑human agreement by subject category and canvas size, separating 16–32 px canvases from those 64 px and above. The manuscript already suggests stronger agreement for larger, concrete subjects, so we would like supporting numerical breakdowns. Third, under a doubled code write-and-edit budget, check whether mid-tier open-weight models narrow the gap against GPT-6 Astra or if the performance difference remains roughly unchanged.

6 comments

Response to Reviewer ZG59

We thank the reviewer for the precise and actionable suggestions. We respond to each point below.

(1/4) W1 + Q3. Are the claims broader than the setup, and would a doubled budget let mid-tier models catch up?

Reviewer's concern. The claims read broader than the setup: what is measured is the ability to drive this drawing library under this budget, which is some distance from pixel art in general. The reviewer asks for at least one stress test, such as a richer or poorer API or a doubled edit budget, especially for open-weight agents that may converge more slowly.

Reviewer's question. "With 2× write/edit budget, do mid-tier open models close on GPT-6 Astra, or does the gap stay?"

Response. We thank the reviewer for the concrete stress tests. We examined the gap from three sides, and in each it stays: even if every artwork that used up the write budget reached GPT-6 Astra's score on its task, no configuration would catch up; GLM-5.3 Flash, a mid-tier open-weight model, still trails GPT-6 Astra by 8.8 points at its highest reasoning effort; and with a poorer interface the gap widens. We also agree that the claims should match the setup: the revision scopes them to pixel-art creation under one programmatic protocol, states in Section 7 that there is a single environment, and leaves comparisons in other environments and agent frameworks to future work.

1. A larger write budget would not let the other configurations catch up. Even if every artwork that used up the write budget reached GPT-6 Astra's score on the same task, no configuration would catch up with GPT-6 Astra: none would gain more than 4.5 points, and GLM-5.3 Flash, a mid-tier open-weight model, would gain 3.8 points and still trail by about 5. The write budget is 25 writes for a generation task and 15 for an editing task. An artwork that stopped short of the budget was submitted by the model's own choice, so more writes could change only the artworks that used the budget up. GPT-6 Astra and GPT-5.6 Sol never do; such artworks come from the models that revise as they go. The table raises each of them to GPT-6 Astra's score on its task:

ConfigurationArtworks that used up the budgetGain if they reach GPT-6 Astra's scoreGap to GPT-6 Astra
Claude Opus 56+0.95.3
Qwen3.8 Max7+0.95.7
Gemini 3.8 Flash25+3.46.4
GPT-5.6 Sol006.8
DeepSeek V4.1 Flash32+4.58.1
GLM-5.3 Flash19+3.88.8
Kimi K33+1.09.3
Gemini 3.1 Pro Preview10+1.714.2
DeepSeek V4 Pro9+3.016.4
DeepSeek V4 Flash8+3.518.2
Qwen3.5-397B-A17B3+1.128.2

The behavior of the models also makes a change of conclusion unlikely. The artworks that used the full budget sit on tasks their models find hard: on the same task they trail GPT-6 Astra by 19.2 points on average, against 11.1 for the same models' other artworks. Comparing the canvas after every run of these 122 artworks, we find that the long processes do not converge steadily: on average 41% of the runs in the second half change nothing; 30% of the artworks return at some point to an earlier canvas; and 42% are still being redrawn after the midpoint, with more than half of their pixels different from the midpoint canvas, while only a third change fewer than 10% of their pixels after it. More writes would mainly prolong this back and forth.

(2/4) W1 + Q3 (continued)

2. More reasoning. We ran three models at further reasoning-effort levels on all 100 tasks: GPT-6 Astra at low, medium, high, and xhigh; DeepSeek V4 Flash with reasoning switched off and at low, high, and max; and GLM-5.3 Flash at low, high, and max. GLM-5.3 Flash rises at every level (62.6, 69.8, 75.0), yet at max it still trails GPT-6 Astra by 8.8 points; DeepSeek V4 Flash saturates at high, with max adding only 0.6. Conversely, GPT-6 Astra at its lowest level, low (81.8), is still above every other configuration. Figure 10b of the revision shows each model's levels side by side on six randomly drawn tasks, and we invite the reviewer to take a look.

3. A poorer API: direct answering. As the poorest possible interface, we designed a direct-answer protocol. In one response, the model declares a palette with one character per color and writes every pixel row at the native canvas size, frame by frame for multi-frame tasks, with the original of an editing task given in the same format. There are no tool calls, no code execution, no view of the result, and no second turn. The answer is parsed into a pixel image and scored by the same judge, and a malformed answer, or one that exceeds the token limit, counts as a failure.

ConfigurationTasksDirect answerAgent
DeepSeek V4 FlashGeneration17.166.9
DeepSeek V4 FlashEditing23.061.4
GPT-6 AstraGeneration, 32×32 and below80.881.4
GPT-6 AstraGeneration, above 64 pixels69.183.1
GPT-6 AstraEditing78.188.0

With the poorer interface the two models keep their order and the gap widens sharply, from 83.8 against 65.5 overall to 76.9 against 18.6. The weaker model can hardly complete the tasks through this interface, and the stronger one drops clearly on large canvases. Appendix J.2 (Table 10) of the revision gives the results by canvas size.

(3/4) W2 + Q1. Are the reference images in the ranking interface a confound?

Reviewer's concern. For generation tasks, the ranking interface shows a GPT-Image-2 rendering of the same brief, so the raters may end up ranking proximity to that look. A no-reference rerank on a subset would show how much the human order moves.

Reviewer's question. "On 20 generation tasks, how much does mean human rank change without the GPT-Image-2 reference?"

Response. Thank you for this suggestion, which we built into the new human rating. Without the reference, the human order barely moves: on the 75 generation tasks, a configuration's mean rank changes by 0.35 places on average, and the leaderboards of the raters with and without references correlate at Spearman ρ = 0.965, with the same first place.

We ran a fuller version than requested: two of the ten raters ranked all 75 generation tasks, and all 25 editing tasks, without any reference. The numbers above compare the twelve configurations' mean ranks from the eight raters with references and from the two without.

For comparison, there are 45 ways to split the ten raters into two and eight. The other 44 splits give generation leaderboards that correlate at 0.930–0.993, and the split by reference falls inside this range.

Appendix I of the revision reports this comparison, together with the judge's agreement with each group of raters.

Reviewer_463VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce PixelBench, a benchmark that formalizes pixel-art generation (75 tasks) and instruction-guided editing (25 tasks) for language-model agents. Tasks specify a written requirement and a native canvas size (with a color ceiling); generation starts from an empty canvas, while editing starts from an existing artwork. They provide a creation environment in which an agent writes a Python program against a pixel-art drawing API, executes it, views the rendered result, revises the source, and iterates until formal submission. Quality is assessed via complete rankings of all 1,200 artworks by four human raters, together with a multimodal judge that is calibrated against those rankings. In the reported experiments, GPT-6 Astra obtains both the highest average judge score and the best average human rank.

Critical strengths

1.The benchmark design is comprehensive and the supporting evaluation is carefully constructed. The authors supply a diverse set of generation and editing tasks at production-relevant canvas sizes, a fully specified agentic creation loop that yields reproducible, programmatically determined artworks, and a human-ranking protocol covering every submission. They further quantify the contribution of presentation choices, rubric, and reference images to judge–human agreement (including leave-one-out comparisons that place the calibrated judge inside the range of inter-rater agreement). These elements collectively demonstrate the reliability and trustworthiness of the evaluation protocol.

Critical weaknesses
  1. The unique or complementary capability being measured is limited. PixelBench primarily exercises agentic tool use, code generation, semantic understanding, and spatial reasoning—capabilities already covered by a range of existing dedicated benchmarks. Consequently, the incremental evaluative value of the proposed suite, relative to those benchmarks, remains modest.
  2. Restricting evaluation to the agentic (program-writing) mode narrows and potentially biases the assessment of pixel-art capability. If the core goal is to measure models’ ability to produce or edit pixel art, an agent-only (especially on a single environment) setting is not the only viable protocol; it places heavier emphasis on agentic scaffolding than on the visual medium itself.
Questions for authors
  1. What distinctive capability of language models does PixelBench evaluate that is not already captured by existing agentic, coding, or spatial-reasoning benchmarks? In other words, why should the community adopt this particular suite?
  2. Beyond the aggregate numerical scores, can the authors provide deeper analysis of the results—identifying characteristic failure modes, qualitatively distinctive behaviors, or concrete implications that could guide future model improvements?
9 comments

Response to Reviewer 463V

We thank the reviewer for the positive assessment of the benchmark design and the evaluation protocol, and for two questions that sharpened both the positioning and the analysis. We respond to each point below.

(1/3) W1 + Q1. What does PixelBench evaluate that existing benchmarks do not?

Reviewer's concern. PixelBench mainly exercises agentic tool use, code generation, semantic understanding, and spatial reasoning, capabilities already covered by a range of dedicated benchmarks, so its incremental value relative to them is modest.

Reviewer's question. "What distinctive capability of language models does PixelBench evaluate that is not already captured by existing agentic, coding, or spatial-reasoning benchmarks? In other words, why should the community adopt this particular suite?"

Response. We thank the reviewer for pressing on this point, the central question for any new benchmark. PixelBench measures whether a model can combine skills that existing benchmarks test one by one into a real production task: creating and editing pixel art under a fixed resolution and a hard palette ceiling, with a quality that no automatic verifier can check. No existing benchmark covers this combination, current approaches cannot complete it directly, and the benchmark comes with a judge validated against complete human rankings.

A composite, real scenario. Pixel art is an asset format that games and digital products use in production. Each skill the task draws on, writing code, using tools, spatial reasoning, and understanding the request, can indeed be measured on its own by existing benchmarks. The scenario requires all of them together, and it also requires editing existing pixel art while keeping its style. SWE-bench, OSWorld, and WebArena play this role in their domains: the component skills are benchmarked elsewhere, and a scenario-oriented benchmark asks whether a model can combine them to complete a real task.

Existing benchmarks each cover a part of it:

  • Code and agent benchmarks test functional correctness that can be verified automatically, such as passing tests or completing a task.
  • Spatial-reasoning benchmarks test understanding and question answering.
  • Image-generation benchmarks target diffusion models, whose outputs are continuous-tone images.
  • Benchmarks for SVG or drawing code written by language models score image-to-SVG reconstruction and SVG editing against a target image or SVG (e.g., StarVector, VectorGym). Open-ended text-to-SVG and drawing tasks are mostly scored with CLIP-style metrics or a multimodal judge for semantic alignment and accuracy, and rarely for aesthetic quality (e.g., SVG-Score, AM-Bench).
  • The existing evaluation of pixel art by language models (Pixel Art Bench) generates once on a 24×24 canvas and scores with a pairwise judge that has not been validated against humans.

Current approaches cannot complete this task directly. General image-generation models cannot meet the native canvas and palette constraints, and answering pixel by pixel without tools largely fails as the canvas grows: DeepSeek V4 Flash answering directly scores 17.1 on generation, against 66.9 as an agent, and completes none of the 14 tasks of 128×128 and above.

The evaluation protocol is our second contribution. Open-ended pixel-art creation has no ground-truth answer, and our protocol needs none. Complete rankings of all 12 artworks of each task by several raters calibrate a multimodal judge that scores one artwork at a time. The protocol was frozen before the external rating began, so the new human data test it independently, and a new model joins the same leaderboard with 100 judge calls. Evaluation without ground-truth answers and judges validated against humans both appear in prior work (e.g., SVG-Score, ImagenWorld); PixelBench evaluates the overall creative quality of pixel art, visual quality together with instruction following, and calibrates its judge against complete per-task rankings.

PixelBench also measures skill specific to pixel art. On canvases of 16–32 pixels, geometric primitives hardly help and quality depends on the placement of each pixel, yet the human raters still separate the models consistently; our response to Reviewer Y5Bg (W1 + Q1) gives the evidence.

The revision scopes its claims to pixel-art creation under one programmatic protocol (Section 7).

(2/3) W2. Does evaluating only the agentic mode, in a single environment, measure the scaffolding?

Reviewer's concern. Restricting evaluation to the agentic, program-writing mode narrows, and may bias, the assessment of pixel-art capability. If the goal is to measure a model's ability to produce or edit pixel art, an agent-only protocol, especially in a single environment, is not the only viable one, and it emphasizes agentic scaffolding more than the visual medium itself.

Response. To address this concern, we added a non-agentic protocol: a direct-answer baseline in which the model writes every pixel in one response, with no tools. It shows why programs are needed at real canvas sizes: direct answering works on small canvases and fails as the canvas grows. Within the agentic protocol, drawing ability carries much of the score: rank does not follow the number of writes or the amount of reasoning, the scaffolding itself rarely causes failures, and our answer to Q2 shows that the lower-ranked models write working programs and fall short in drawing.

The direct-answer baseline. In one response the model declares a palette with one character per color and then writes every pixel row at the native canvas size, frame by frame for multi-frame tasks, with the original of an editing task given in the same format. There are no tool calls, no code execution, and no view of the result; the answer is parsed into a pixel image and scored by the same judge.

Direct answering gets harder as the canvas grows. DeepSeek V4 Flash delivers a valid artwork on 25 of the 28 tasks of 32×32 and below, and on none of the 14 tasks of 128×128 and above; its generation score is 17.1, against 66.9 as an agent. The strongest model, GPT-6 Astra, matches its agent score on small canvases (80.8 against 81.4 at 32×32 and below) and falls clearly behind above 64 pixels (69.1 against 83.1).

(3/3) Q2. Can the authors provide deeper analysis beyond the aggregate scores?

Reviewer's concern. The submission mainly reports aggregate scores, and its analysis of the results is limited.

Reviewer's question. "Beyond the aggregate numerical scores, can the authors provide deeper analysis of the results—identifying characteristic failure modes, qualitatively distinctive behaviors, or concrete implications that could guide future model improvements?"

Response. We thank the reviewer for this suggestion, which led to a new analysis in Appendix L. Its main finding is that the lower-ranked configurations write working programs and fall short in drawing: they assemble subjects from generic geometric shapes and render little light, volume, or detail. The analysis compares the programs and artworks of the four configurations the human raters rank highest (GPT-6 Astra, Gemini 3.8 Flash, Claude Opus 5, Qwen3.8 Max) with those of the four they rank lowest (Gemini 3.1 Pro Preview, DeepSeek V4 Pro, DeepSeek V4 Flash, Qwen3.5-397B-A17B).

The lower-ranked configurations do not fail at programming. Their program executions fail about as often (5.4% against 4.9%), and 399 of their 400 artworks were submitted formally, against 397. Within a task, the number of failed executions is unrelated to human rank (mean Spearman ρ = −0.01 [−0.08, 0.06]), and among artworks whose programs never failed, the lower four trail by 4.3 places of mean rank, the same gap as over all artworks.

They fall short in drawing. Their programs assemble subjects largely from generic geometric primitives: rectangles, circles, and ellipses produce 48% of their drawn pixels, against 30%. Their artworks render less light, volume, and detail: 22% of their pixels lie inside flat single-color areas, against 12%, and within a task more flat area goes with a worse human rank (mean Spearman ρ = −0.49 [−0.55, −0.42]). In a review of 16 tasks drawn at random, the lower-ranked artwork of each task's best and worst pair shows a misconstructed or unreadable main shape in 10 cases, flat unshaded fills in 6, and weak composition, such as objects set side by side without overlap or a focal point, in 4. Pixel-art techniques such as dithering, outlines, and symmetry are equally common in both groups.

Behavior. GPT-6 Astra and GPT-5.6 Sol write their program about twice per task and usually compose in one pass, while several configurations revise 11 to 15 times; more iteration does not go with better artworks. Editing depends on seeing the original: without image observation, GPT-6 Astra's editing score falls from 88.0 to 77.8.

Implications. Every configuration can write a working program; what separates them are visual decisions: how the form of a subject is constructed, how light and volume are rendered, how the canvas is composed, and where detail is spent. Progress on PixelBench therefore depends on these drawing abilities.

Reviewer_Y5BgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a benchmark for evaluating LLMs on pixel-art generation and editing them through an agentic Python drawing environment. It also defines 75 text-to-image tasks and 25 editing tasks, covering different canvas sizes, visual categories, and editing operations. The evaluation includes 12 model configurations, collecting human rankings from four raters, and introducing a multimodal LLM judge calibrated against these rankings.

Critical strengths
  1. The paper introduces pixel-art creation and editing through a benchmark called PixelBench to evaluate state-of-the-art LLM performance on this task.

  2. The benchmark includes 75 text-to-image tasks and 25 editing tasks to perform a fine-grained evaluation of where exactly the LLM falls short.

  3. The paper also obtains 12 model configurations. It collects 1,200 generated artworks and creates a drawing API that provides a common environment for evaluating tool-using LLMs, making comparisons more systematic and the benchmark easier to reproduce.

Critical weaknesses
  1. The proposed environment primarily asks models to generate Python programs where the primitives are polygons, SVG paths, Bézier curves, circles, and rectangles available already. Then how is this task different from primitive-based SVG generation? There is no comparison between a model using high-level geometric primitives and one required to make explicit pixel-level decisions.

  2. The human evaluation is insufficiently established as a reliable ground truth. Only four human raters participate. The number is so small as to establish a new benchmark. Also, it is not reported who they are by profession. It is hard to rely on that evaluation without any further information.

  3. The primary evaluator is GPT-6 Astra, which is also one of the evaluated systems and obtains the highest leaderboard position. This is obvious and makes the whole evaluation protocol invalid. The authors provide an additional experiment using Claude Opus 5 as the judge. I appreciate this experiment. There is a high possibility that both evaluators may share systematic preferences for particular visual styles, compositional conventions, or image-generation characteristics. (Ultimately, they are all closed-source LLMs.)

  4. The final judge–human Kendall correlation of 0.5583 indicates moderate agreement, leaving substantial disagreement in artwork rankings. More importantly, agreement varies considerably across categories, reaching 0.685 for scenes but dropping to 0.147 for materials. Although human raters also show low agreement on materials (0.194), this raises concerns about the reliability and consistency of the proposed evaluation protocol across different pixel-art categories.

Questions for authors
  1. What evidence establishes that PixelBench measures pixel-art-specific artistic competence rather than general code-generation and visual composition capabilities?

  2. What are the professional qualifications and pixel-art experience of the four human raters?

  3. Why should the overall leaderboard be considered reliable when judge–human agreement is extremely low for some benchmark categories?

5 comments

Response to Reviewer Y5Bg

We thank the reviewer for questions that go to the foundations of the benchmark; answering them made the positioning of PixelBench and its evidence clearer. We respond to each point below.

(1/5) W1 + Q1. How does PixelBench differ from primitive-based SVG generation, and does it measure pixel-art-specific skill?

Reviewer's concern. The drawing environment provides primitives such as polygons, SVG paths, Bézier curves, circles, and rectangles, and the models mainly write Python programs that call them, which looks much like primitive-based SVG generation. The submission does not compare a model that uses high-level geometric primitives with one that must make explicit pixel-level decisions, so it is unclear whether PixelBench measures pixel-art-specific skill or general code generation and visual composition.

Reviewer's question. "What evidence establishes that PixelBench measures pixel-art-specific artistic competence rather than general code-generation and visual composition capabilities?"

Response. This is a fair question about what the benchmark measures, and we added the comparison the reviewer asks for. In short, PixelBench delivers a native pixel matrix under a palette ceiling, in which every pixel is visible and judged, and the models treat it that way: only 17% of their programs use SVG paths, which draw 3.2% of the pixels, and on canvases of 16 pixels and below nearly half of the artworks are written out pixel by pixel. A new baseline that removes all primitives confirms that the small canvases test pixel-level decisions: deciding every pixel explicitly, GPT-6 Astra matches its agent score at 32×32 and below (80.8 against 81.4), and the human raters separate the models there in an order of their own. In addition, we observe that the models' drawings use a number of techniques specific to pixel art.

Primitives against explicit pixel decisions. The direct-answer baseline works as follows. In one response, the model first declares a palette that assigns a one-character code to each color, then writes one row of codes per pixel row at the native canvas size, frame by frame for multi-frame tasks; an editing task gives the original in the same format. There are no tool calls, no code execution, no view of the result, and no second turn. The answer is parsed into a pixel image and scored by the same judge, and a malformed answer, or one that exceeds the token limit, counts as a failure.

This setting removes the primitives entirely and forces an explicit decision for every pixel. On canvases of 32×32 and below, GPT-6 Astra answering directly scores 80.8, almost the same as the agent with primitives (81.4); above 64 pixels it drops to 69.1 (agent: 83.1). On small canvases, explicit pixel decisions are enough for GPT-6 Astra to match the agent; as the canvas grows, primitives and programs help the model organize the overall structure.

The human rankings separate the two kinds of skill as well. On the 28 generation tasks of 16–32 pixels, the ten human raters still separate the models consistently: two random halves of the raters order the configurations alike (Spearman ρ = 0.97). This order differs from the one on the 42 tasks of 64 pixels and above: the two correlate at 0.87, against 0.97–0.99 between rater halves within either band. The two leaderboards share the first two places, GPT-6 Astra and Gemini 3.8 Flash, and the last three, and reorder the middle: DeepSeek V4.1 Flash is ninth on small canvases and fifth on large ones, while Gemini 3.1 Pro Preview moves from sixth to ninth and GPT-5.6 Sol from fourth to sixth. The small canvases measure pixel-level craft that differs from the structure and composition rewarded on large canvases, and PixelBench covers both.

(2/5) W1 + Q1 (continued)

What the models do. We analyzed the programs that produced the artworks of the new round. Only 17% of them use SVG paths, and path-drawn pixels make up 3.2% of the final images. On canvases of 16 pixels and below no program uses a path, and nearly half of the artworks (48%) are written out in the program as a character pixel map: a list of equal-length strings with one character per pixel, which is how pixel art is drawn by hand.

The artworks also show many techniques specific to pixel art. Each example below comes from a generation task drawn from an empty canvas:

TechniqueArtwork (task, model)What the pixels show
Banded gradienttypography, GPT-6 AstraThe letters are filled with five flat bands from light to dark, with a shadow below them
Ditheringsummer-seaside, Qwen3.8 MaxThe pale yellow of the sun's outer ring alternates pixel by pixel with the sky behind it in a checkerboard, forming a halo
Selective outlinefemale-elf-archer, Gemini 3.8 FlashNo uniform black outline: the cloak, the leg, and the boot are each closed with a darker tone of their own color
Manual anti-aliasingtwo-character-standoff, GPT-6 AstraAt each step of the blade's diagonal edge sits one pixel in a tone between the blade and the background
Hue shiftingforest-slime-monster, GPT-6 AstraThe slime's green ramp shifts toward blue-green in the shadows and toward yellow-green in the highlights
Single-pixel detailfemale-lab-researcher-chibi (16×16), Gemini 3.1 Pro PreviewEach eye is 1×2 pixels, and the blush and the pen in the pocket are one pixel each
Seamless tilingrpg-wooden-planks-material, GPT-6 AstraThe wood grain continues across the tile border and the plank ends are staggered, so a 3×3 repetition shows no seam

These techniques appear in the artworks of several models, which do not simply carry over the way they generate SVG. Figure 12 in Appendix K of the revision shows enlargements of all seven, and we invite the reviewer to take a look.

Deliverable and judgment. SVG generation delivers a resolution-independent geometric description: a renderer decides the final pixels, anti-aliases edges automatically, and blends colors freely. PixelBench delivers the native pixel matrix itself. The engine places every drawing call directly on the native grid without anti-aliasing; only palette colors can be used, under a hard ceiling on their number; and the model views, edits, and submits pixels, which the human raters and the judge also assess at the native pixel level. Even a circle() call therefore leaves a staircase of hard-edged pixels, and whether to clean up its edge pixel by pixel remains the model's decision. The smaller the canvas, the less a primitive helps: on a 16×16 canvas, a circle of radius 3 is only seven pixels wide. The rubric also asks that every pixel be placed deliberately at 32 pixels and below.

Finally, when pixel art is created through programs, pixel-art skill is coupled with coding and composition, and we do not claim to separate them completely. Human pixel artists also use primitives such as line, shape, and fill tools, especially on large canvases, and using them well is part of the craft. PixelBench evaluates this complete creative task, and its pixel-art-specific part is secured by the native canvas, the palette ceiling, judgment at the native pixel level, and dedicated small-canvas, material-tiling, and frame-by-frame animation tasks.

(3/5) W2 + Q2. Number and qualifications of the human raters

Reviewer's concern. The human evaluation is not yet established as a reliable ground truth: four raters are too few for a new benchmark, and the submission does not report who they are by profession.

Reviewer's question. "What are the professional qualifications and pixel-art experience of the four human raters?"

Response. We agree that four raters were too few and that their background should have been reported. The four raters of the submission were members of the research team; three had years of art experience or experience in pixel-style game production, and one had little related experience. To give the human ground truth a firmer basis, the revision replaces them with ten external raters with professional art backgrounds, who produced complete rankings of all 1,200 artworks of the new round; all human data in the revision come from them. Their leaderboard is stable: two random halves of the ten raters give leaderboards that correlate at Spearman ρ = 0.98.

The ten external raters come mainly from art and game design, with experience in graphic design, years of painting, or pixel-style game art. Each ranked all 100 tasks, and two ranked every task without a reference.

In the ranking interface the artworks are anonymized under random labels, and each rater sees them in an independently drawn random initial order. The interface records the initial order and the time spent on each task, and the initial position has little effect on the final rank: the Kendall τ between where an artwork starts and where it is ranked averages 0.10 over raters. Appendix I of the revision describes the raters and shows the interface (Figure 9).

Nine of the ten raters individually rank GPT-6 Astra first, and the tenth ranks it second.

Reviewer_9YSiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PixelBench defines 75 generation and 25 editing tasks on native pixel canvases and supplies a code-based creation environment with execution and revision. Four raters rank 1,200 artworks from 12 model configurations. The paper calibrates an automatic judge through image presentation, task rubrics, and reference images; it then compares model configurations and studies the effect of image observation. Supplementary experiments examine repeat scoring, a second judge, and task slices.

Critical strengths
  1. A concrete, reusable evaluation setting with meaningful constraints. Native resolution and palette limits distinguish the task from generating a high-resolution image in a pixel-like style. The editable program representation, generation/editing split, and full task cards make the benchmark more inspectable than a leaderboard alone.
  2. The judge is validated unusually carefully. Calibration raises Kendall agreement from 0.4507 to 0.5583 (Table 1, p. 7); importantly, the paper reports improvement on 77 tasks outside the development subset. Leave-one-rater-out comparisons put judges and humans on a common reference. Repeat scoring (Appendix D) and a Claude second judge over all 1,200 artworks (Appendix F) address important sources of evaluator instability and self-preference concerns.
Critical weaknesses
  1. The reported uncertainty does not cover stochastic creation or equal inference cost. The benchmark contains one submitted artwork per configuration and task. Appendix D repeats judge scoring of fixed artworks, which estimates evaluator sampling but not the variability of the agents' creative runs. A common cap on source edits also does not equalize tokens, free executions, wall time, or provider-specific default reasoning effort (Sections 4 and 6). Consequently, the leaderboard is credible for these observed configurations, but stronger claims about underlying model capability or close comparisons require more evidence.
  2. Human alignment is conditional on a small, reference-anchored task distribution. Human raters and judges share reference images, so agreement is measured within that protocol rather than against unanchored preferences. Only two multiview and five material tasks are present; material judge–human agreement is 0.147 (Appendix G). These slices limit broad claims about editing or pixel-art quality. The paper appropriately reports a confidence interval crossing zero for the judge–human difference; this is not an equivalence test and should not be interpreted as proving human-level interchangeability.
Questions for authors
  1. Can you repeat the creation of representative leading configurations and report paired uncertainty incorporating both task sampling and creative-run variability? Please distinguish this from the existing repeat-judge experiment.
  2. Can you release per-run tokens, execution counts, elapsed time, and cost, and state whether model ordering is robust under a matched inference budget?
  3. How does calibration behave when raters do not see a reference, or when references come from a different source? Please either report this sensitivity or explicitly scope human alignment to the shared-reference protocol.
4 comments

Response to Reviewer 9YSi

We thank the reviewer for the careful reading and for three concrete requests, each of which we turned into a new experiment. We respond to each point below.

(1/3) W1 + Q1. Can repeated creation bring the randomness of creation into the uncertainty?

Reviewer's concern. Each configuration submits one artwork per task. Appendix D of the submission repeats the judge's scoring of fixed artworks, which estimates the judge's sampling but not the variability of the agents' creative runs. The leaderboard is therefore credible for the observed configurations, while stronger claims about model capability, or comparisons of close configurations, need more evidence.

Reviewer's question. "Can you repeat the creation of representative leading configurations and report paired uncertainty incorporating both task sampling and creative-run variability? Please distinguish this from the existing repeat-judge experiment."

Response. We agree that judge repeats alone do not capture the randomness of creation, and we ran the experiment the reviewer describes. Three configurations created all 100 tasks three times from scratch, and resampling tasks and creation runs together gives paired intervals that include both sources of variation: GPT-6 Astra leads GPT-5.6 Sol by 5.8 points [4.2, 7.4], and GPT-5.6 Sol leads DeepSeek V4 Flash by 12.1 [9.3, 14.9], far beyond the run-to-run variation of creation.

The three configurations differ in position and kind: GPT-6 Astra, first on the leaderboard; GPT-5.6 Sol, from its middle; and DeepSeek V4 Flash, an open-weight model we serve ourselves. Every run used identical settings, and the same judge scored every artwork.

ConfigurationMean score, runs 1 / 2 / 3SD of the meansPer-task SD across runsCreation SD
GPT-6 Astra83.8 / 83.0 / 81.71.04.43.7
GPT-5.6 Sol76.9 / 77.4 / 76.70.35.55.0
DeepSeek V4 Flash65.5 / 64.0 / 65.40.910.09.8

The creation SD removes the judge's own repeat-scoring variance, measured in Appendix D (pooled SD 2.30), from the variance of a task's score across runs. What remains comes from creation itself, which separates the randomness of creation from that of the judge.

Across the three runs, the 100-task means move by at most about two points. On a single task, the variability of creation depends on the model: 3.7 points for GPT-6 Astra, 5.0 for GPT-5.6 Sol, and 9.8 for DeepSeek V4 Flash, so the stronger models create more consistently.

Close configurations. We agree that a single creation run cannot settle the order of configurations a point or two apart. Estimated from these runs, with one run per configuration and 100 tasks, differences smaller than about 1.6 points cannot be resolved; three runs narrow this to about 1.1 points, and no number of runs narrows it below about 0.8 points, the limit set by the number of tasks. Separating configurations closer than that would take more tasks, which repeated creation cannot replace. Most conclusions of the paper rest on much larger gaps. By judge score, the configurations form four groups: GPT-6 Astra (83.8); seven configurations between 74.5 and 78.5; three between 65.5 and 69.5; and Qwen3.5-397B-A17B (55.6). The groups are 5 to 10 points apart, and the paper draws no conclusion about order within a group (Section 6). The main table still uses run 1, the artworks the human raters ranked. Appendix J.1 (Table 8) gives the full results, and Figure 10a shows the three runs side by side on six randomly drawn tasks; we invite the reviewer to take a look.

(2/3) W1 + Q2. Can the per-run costs be released, and is the ordering robust under a matched inference budget?

Reviewer's concern. The configurations share only a cap on source edits; tokens, free executions, wall time, and each configuration's reasoning effort are not equalized (Sections 4 and 6 of the submission). The leaderboard thus compares these specific configurations, and whether it extends to model comparisons at matched budgets is unclear.

Reviewer's question. "Can you release per-run tokens, execution counts, elapsed time, and cost, and state whether model ordering is robust under a matched inference budget?"

Response. Yes. Every model call of the new round is logged and will be released, and the leader's position holds as the inference budget changes: GPT-6 Astra uses the fewest output tokens of any configuration and stays above every other configuration at every reasoning effort, including its lowest.

What is logged. The logs contain input, output, and reasoning tokens with cache hits; the number of calls and of source writes; and latency and retries. We also convert each task to cost at the providers' list prices. Appendix J.3 of the revision reports the per-task averages of all of these (Table 11) and plots score against cost (Figure 11). We release wall-clock time but do not compare it, since it depends mostly on the provider's load at the time; running the programs the models write takes a negligible share.

Matched budgets. We agree with this observation. We did not match token budgets directly: a hard token cap often cuts a reasoning model off mid-thought, before it can write a complete program, so the measurement would mostly reflect truncation and say little about the model's ability at that budget. We therefore follow the more common practice of comparing each model across reasoning-effort levels, and the cost–score figure lets one compare configurations at equal spending. The results below show that the leader stays ahead as the inference budget changes.

The leader uses the least inference. GPT-6 Astra produces the fewest output tokens of any configuration, about 2.3k per task with reasoning included. It writes its program twice per task on average, finishes in about two minutes, and costs USD 0.24 per task, less than half of what each of the next three configurations on the leaderboard costs (USD 0.49–0.52).

Changing the reasoning effort does not change the leader. We ran three models at further effort levels on all 100 tasks. GPT-6 Astra scores 81.8, 82.4, 83.8, and 84.7 from low to xhigh and stays above every other configuration even at low; xhigh costs about twice as much as high for 0.9 points. GLM-5.3 Flash rises at every level, yet reaches only 75.0 at max. DeepSeek V4 Flash saturates at high. Every level submits formally on at least 94 of the 100 tasks, so the differences come from the artworks.

The write budget rarely binds the leaders. GPT-6 Astra and GPT-5.6 Sol write about twice per task, at most 9 and 7 times, and never use the full budget. Our response to Reviewer ZG59 (W1 + Q3) shows that a larger write budget would not let the other configurations catch up.

Table 2 of the revision gives the reasoning effort of every configuration, and the leaderboard is stated for these configurations.

(3/3) W2 + Q3. How does calibration behave without a reference, or with references from another source?

Reviewer's concern. The reviewer raises three points. The human raters and the judge see the same reference image, so agreement is measured within that protocol and may not reflect preferences without the reference. The multiview (2 tasks) and material (5 tasks) slices are small, and judge–human agreement on materials was 0.147 (Appendix G of the submission), which limits broad claims about editing or pixel-art quality. Finally, a confidence interval of the judge–human difference that crosses zero is not an equivalence test and does not show that the judge and humans are interchangeable.

Reviewer's question. "How does calibration behave when raters do not see a reference, or when references come from a different source? Please either report this sensitivity or explicitly scope human alignment to the shared-reference protocol."

Response. Thank you for this question, which shaped the design of the new human rating. Two of the ten new raters ranked all 100 tasks without any reference, and the human leaderboard barely changes: the leaderboards of the raters with and without references correlate at Spearman ρ = 0.972, with the same first place. On the judge side, adding the reference raises agreement by a similar amount for both groups of raters. For references from other sources, we take the reviewer's second option and scope the judge–human agreement to the shared-reference protocol (Section 5.3), and the revision now reports the judge's position among the raters: under leave-one-out, the judge lies inside their range and above three of the ten.

Human side. For the new round we invited ten raters with professional art backgrounds to rank all 1,200 artworks; the two without a reference saw no GPT-Image-2 rendering for generation and only the original for editing. Taking the mean ranks of the eight raters with references and of the two without as two separate leaderboards, every configuration's mean rank moves by less than one place between them.

Judge side. The calibration ladder (Table 1) contains a version without the reference image, the rubric step before the reference is added. On the new round:

Judge versionTen-rater consensusEight raters with referenceTwo raters without reference
Without reference0.5440.5360.504
With reference (final)0.5570.5490.514

Adding the reference changes agreement little, and by a similar amount for both groups of raters; the reference does not pull the judge toward the raters who saw it.

Small slices. Under the paper's leave-one-out comparison (hold out one rater and take the mean rank of the other nine as the consensus), the judge falls within the range of the ten raters on every slice. On materials, which the reviewer mentions, the judge reaches 0.453, above seven of the ten raters; on editing it reaches 0.535, within the raters' range of 0.492–0.674. Our response to Reviewer ZG59 (Q2) gives the slice table, and Appendix G (Table 7) of the revision covers every subject, canvas size, and editing type.

Equivalence. We agree that an interval crossing zero does not by itself show that the judge and humans are interchangeable, and the revision reports the judge's position among the raters directly. Under leave-one-out, the judge reaches 0.554, inside the range of the ten raters (0.465–0.649) and above three of them. The paired difference between the judge and a held-out rater is −0.023 [−0.056, +0.008]; even at the far end of this interval (0.056), the judge trails the average rater by about half as much as the lowest rater does (0.113). Replacing one human rater with the judge therefore keeps the quality of the ranking within the normal range of human raters.

On this basis, the community can evaluate new models on PixelBench with the calibrated judge. We release all human rankings, so anyone can re-check the judge with the same method.

Dear Reviewer 9YSi,

As the discussion period closes today, we would like to briefly recap the new experiments that your three questions led to. For Q1, three configurations each created all 100 tasks three times, and paired intervals that include both task sampling and creation variability keep the gaps clear: GPT-6 Astra leads GPT-5.6 Sol by 5.8 points [4.2, 7.4] (Section 6.3, Appendix J.1). For Q2, we release per-run tokens, calls, time, and cost; GPT-6 Astra uses the fewest output tokens of any configuration, costs less than half as much per task as the next three, and stays first at every reasoning effort, including its lowest (Appendices J.2 and J.3). For Q3, two of the ten new raters ranked all tasks without a reference, and the human leaderboard barely changes (Spearman ρ = 0.972, same first place); the judge–human agreement is scoped to the shared-reference protocol, and under leave-one-out the judge falls within the raters' range on every slice (Section 5.3, Appendices G and I).

We would be glad to clarify anything before the discussion closes, and if these results address your concerns, we would be grateful if they could be reflected in your final assessment.

Best regards, The Authors

Comments on the paper

1 comment

General response to all reviewers

We thank the four reviewers for their careful reading and constructive comments. PixelBench asks language models to write programs on native pixel canvases to create pixel art (75 generation tasks) and to edit it (25 editing tasks); a multimodal judge, calibrated against complete human rankings, scores each artwork.

Summary. Following the reviews, we reran the whole evaluation with complete logging, had ten external raters with professional art backgrounds rank all 1,200 new artworks, and added the experiments the reviewers asked for. The new round supports the paper's conclusions: nine of the ten raters individually rank GPT-6 Astra first and the tenth ranks it second; the judge's agreement with the human consensus falls within the range of the ten raters; and the judge and human leaderboards correlate at Spearman ρ = 0.958. The changes in detail:

  1. A complete new evaluation round. We reran every configuration under the released protocol (the new round below). The first round did not log every model call in full, and information such as token usage was missing; to report results that are consistent and complete, we reran everything. Every call of the new round records its tokens, time, cost, and retries, and these logs will be released with the code. All experimental results, human data, and calibration tables in the revision come from the new round, so the scores of individual configurations differ from the submission. The main table now includes DeepSeek V4.1 Flash, and GPT-6 Astra without image observation is reported among the analyses scored by the judge (Section 6.3).
  2. External human rating. Ten external raters with professional art backgrounds produced complete rankings of all 1,200 artworks of the new round; two of them ranked every task without any reference image.
  3. New experiments and analyses: repeated creation (Appendix J.1); reasoning effort and a direct-answer baseline without tools (Appendix J.2); cost per task (Appendix J.3); the second judge rerun on the new round with Claude Opus 5, together with the open-weight DeepSeek V4.1 Flash as a further judge (Appendix F); pixel art compared with SVG generation (Appendix K); and creation behavior and failure modes (Appendix L).

Section, appendix, table, and figure numbers in our responses refer to the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.