acceptodds
Under review as a conference paper at ICLR 2027

PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

Abstract

Figural divergent thinking is the ability to develop a given shape fragment into an original drawing. In humans, this ability is assessed with incomplete-drawing tasks. We introduce PainterBench, a benchmark that ports the incomplete-drawing task to the agentic setting. The agent draws on a canvas through tool calls and observes the result after every turn. The canvas includes a starting shape which cannot be erased, and the agent's goal is to incorporate this shape into the most original drawing it can produce. The task is open-ended, and the agent itself decides when the drawing is finished. The benchmark tests incremental visual planning over a short horizon and the transfer of creative ability from pretraining to multi-turn tool use. We evaluate 14 multimodal language models from small to frontier scale. Across the primary study and six sensitivity analyses, we collect 2,700 drawings and crowdsource creativity and recognizability ratings for every drawing and for 300 human reference drawings. We also present ViDrA-adapted, an automated scorer that predicts human creativity ratings of agent drawings ( on random held-out test split). Figural divergent thinking varies widely across the 14 models, and GPT-6 Astra produces the most creative drawings. Relative to the human drawings, the agent drawings score higher in creativity but lower in recognizability. We release the final drawings, per-round canvas snapshots, tool call traces, stimulus bank, benchmark harness, crowdsourced ratings (N = 72,000), and ViDrA checkpoint.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, creativity, figural divergent thinking, drawing, tool use, human evaluation, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_i25FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PainterBench ports the MTCI incomplete-figure task to tool-using MLLMs. Agents draw on a canvas with 14 tools, see the result after each turn, and decide when to stop. Fourteen models produce 2,100 drawings, each rated by 12 crowdworkers, with 300 human drawings as reference. The authors show that AuDrA mostly tracks ink on agent drawings and train their own scorer (ViDrA-adapted, r ≈ 0.84). Agents are rated more creative but less recognizable than humans.

Critical strengths
  1. Careful measurement: the rater count comes from a reliability pilot, the ink baseline exposes AuDrA's failure on agent drawings, and the scorer is validated leave-one-model-out.
  2. The released traces and the process analysis, which separates between-model from within-model effects.
Critical weaknesses
  1. The claim that agents are more creative than humans is weakly controlled. Humans and agents saw different stimuli, and freehand drawings differ from geometric primitives: agent drawings sit 1.9–4.3 SD from the human corpus. Creativity also correlates with ink (r = 0.40), and most models use far more ink than humans. How rating batches were composed is not described, which matters under within-rater z-scoring.
  2. The score may not capture divergent thinking. Removing the stimulus barely changes creativity (−0.02), and models keep repeating themes (Luna titles 61% of drawings "cosmic", Astra draws a snail 24% of the time), which per-drawing ratings ignore.
Questions for authors
  1. How were rating batches composed, and does the human gap hold after controlling for ink?
  2. Can you report a diversity-adjusted model ranking?
  3. Why does the context manipulation use non-adapted ViDrA instead of the human ratings you collected?
1 comment
  1. Composition of rating batches:

We have added a description of how the rating batches were composed to appendix G:

"The crowd workers rated the images in batches of 30 images. Drawings were assigned to batches stratified by source (main study, sensitivity analysis, human reference), so each batch contained roughly the same proportions of the full set of images. This mix is also what the per-rater z-scoring relies on. Each batch was rated by 12 distinct raters. The drawings were randomly shuffled within the batch when the batch was created. All 12 raters of a batch then saw the drawings in the same order. Image filenames were hashed to not reveal the drawing's source."

  1. Diversity:

An important comment about diversity in ideas across models. Our analysis is mostly focused on rating individual images. Divergent thinking, here, is used as a term to denote the ability to create something novel and effective (recognizable). Examining homogenization across independent trials was a side quest that was worth reporting in Section 4.3 as one observed failure mode, but it is not a strong focus of the paper. There has been work on idea diversity in LLMs, such as Artificial Hivemind (arXiv:2510.22954), and we will gladly acknowledge this prior work in the paper. Further, Table 2 is already quite crowded, and reporting diversity-adjusted scores would only make the table less understandable. A deeper analysis of idea diversity between models is left to future work. Same goes for within-family variability and within-model variability. We collected five images per stimulus and model, and variability is clearly visible in the snapshots on Hugging Face. Variability in outputs of tool-calling models in creative tasks over long horizons is worth investigating further in future work.

  1. Human ratings in context manipulation:

The reviewer is correct to point this out. The other sensitivity analyses draw on the collected human ratings, and so should the context ablation. We now use the human ratings in this analysis, not the ViDrA scores.

Reviewer_w9mjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose PainterBench to devaluate figural divergent thinking in tool-using LLMs. Agents complete an initial shape through drawing operations, receive visual feedback after each turn, and decide when to finish. The paper evaluates 14 models, collects human creativity and recognizability ratings, analyzes drawing processes, and adapts an automated scorer, ViDrA. The results reveal substantial model differences, limited revision, and higher creativity but lower recognizability ratings than the human reference sample.

Critical strengths

(1) A useful benchmark contribution. Adapting an established incomplete-drawing task to multi-turn tool use extends creativity evaluation beyond text generation and reference-matching tasks.

(2) Substantial human evaluation. The study collects 72,000 ratings with 12 raters per drawing and reports rating reliability. Creativity and recognizability are assessed separately, yielding more informative results than a single overall score.

(3) Strong supporting resources and analyses. Drawing traces and canvas snapshots expose the production process. ViDrA-adapted is validated on held-out models and stimuli, and the analysis identifies meaningful failures such as repeated ideas.

Critical weaknesses

(1) Sensitivity checks cover only one model. All six analyses use GPT-5.6 Luna, and most conditions have only one trial per stimulus. This does not establish whether interface, instruction, or context changes affect stronger and weaker models differently. Testing every model is unnecessary, but repeating the main checks on roughly three representative models—one frontier, one middle, and one lower-capability model—would make the robustness claims more convincing.

(2) Some sensitivity results report recognizability without the corresponding creativity outcome. Although both ratings were collected, the oracle and framing analyses mainly report recognizability improvements. These changes could make drawings easier to recognize while reducing originality. Both outcomes should be presented together.

(3) The human comparison needs a limited interpretation. The reference drawings come from an existing AuDrA corpus rather than humans completing the same 30 stimuli under comparable task conditions. The shared rating procedure is valuable, but higher agent ratings should be interpreted as a comparison with this reference sample, rather than evidence of generally superior human-level creativity.

Questions for authors

(1) Can you repeat the main sensitivity checks on about three models representing frontier, middle, and lower capability? Do the conclusions and relative rankings remain similar?

(2) For the oracle and framing conditions, can you report both creativity and recognizability? Does recognizability improve without reducing creativity?

(3) Can you also check the context results using human ratings or ViDrA-adapted?

1 comment
  1. Extending the sensitivity analysis to three models:

The reviewer asks to extend the sensitivity analysis to three models (from one model). We will extend the analysis to three models as suggested: a frontier, middle, and lower capability model from different model families (per new model: 390 images, 13 batches, 156 crowd workers). Note that because this involves preparation and human evaluation, we could not implement this change in time for the rebuttal revision. The paper and Hugging Face repository will be amended with the new results and data.

  1. Reporting in the sensitivity analyses:

Some sensitivity analyses reported only one variable, because the main purpose of the respective analysis was to analyze that variable. The questions in Table 3 show which variable is our focus for that manipulation. But it is true that the other variable should be reported for sake of completeness. Note that reporting both variables in the main text would negatively affect readability. For this reason, we have extended Appendix E and report results there.

  1. See our comment on reviewer i25F, #3.
Reviewer_R6hcOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PainterBench, a benchmark designed to measure whether multimodal language models can demonstrate visual creativity while operating as tool-using agents. Painter bench gives model incomplete drawings and ask it to complete the drawing into an original work. The agent must draw incrementally using geometric tools, inspect the canvas after each turn, and decide when its drawing is finished.

Critical strengths
  • A very interesting benchmark and interesting findings, claiming that AI drawings receive higher creativity scores.
Critical weaknesses
  • The human comparison drawings come from the existing AuDrA corpus, whereas the LLM drawings use a newly generated set of 30 stimuli. The AI and human drawings are not responses to identical starting shapes under identical production interfaces, and I believe this would have a significant impact on the results and findings.
Questions for authors
  • Is it possible to have equivalent comparisons with the same starting shapes between humans and LLMs
2 comments

I would prefer the authors to respond to rebuttal questions. My original assessment is around 2.5.

Comparing agents with humans, mismatch in stimuli:

The PainterBench stimuli are different from AuDrA's stimuli. The reviewer rightfully pointed out that this creates issues when comparing agent drawings with human drawings. We will conduct a new stimulus-matched comparison. To this end, we will produce 532 new agent drawings from AuDrA stimuli (14 models x 38 drawings per model, allocated in proportion to the AuDrA stimuli used in the sample of 300 human reference drawings). We will get these drawings rated by 12 crowd workers per image, using the same approach and rating protocols as the main study. A comparison of the new agent drawings with the already rated 300 human reference drawings will settle the open concerns about the stimulus mismatch. Every statement in the paper that compares agent and human drawings will be based on the stimulus-matched comparison, whatever its outcome. This also refers to statements about distance from the human corpus. Note that because this involves preparation and human evaluation, we could not implement this change in the uploaded rebuttal revision. Data collection is ongoing, and the paper and Hugging Face repository will be amended with the new data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.