- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PainterBench ports the MTCI incomplete-figure task to tool-using MLLMs. Agents draw on a canvas with 14 tools, see the result after each turn, and decide when to stop. Fourteen models produce 2,100 drawings, each rated by 12 crowdworkers, with 300 human drawings as reference. The authors show that AuDrA mostly tracks ink on agent drawings and train their own scorer (ViDrA-adapted, r ≈ 0.84). Agents are rated more creative but less recognizable than humans.
Critical strengths
- Careful measurement: the rater count comes from a reliability pilot, the ink baseline exposes AuDrA's failure on agent drawings, and the scorer is validated leave-one-model-out.
- The released traces and the process analysis, which separates between-model from within-model effects.
Critical weaknesses
- The claim that agents are more creative than humans is weakly controlled. Humans and agents saw different stimuli, and freehand drawings differ from geometric primitives: agent drawings sit 1.9–4.3 SD from the human corpus. Creativity also correlates with ink (r = 0.40), and most models use far more ink than humans. How rating batches were composed is not described, which matters under within-rater z-scoring.
- The score may not capture divergent thinking. Removing the stimulus barely changes creativity (−0.02), and models keep repeating themes (Luna titles 61% of drawings "cosmic", Astra draws a snail 24% of the time), which per-drawing ratings ignore.
Questions for authors
- How were rating batches composed, and does the human gap hold after controlling for ink?
- Can you report a diversity-adjusted model ranking?
- Why does the context manipulation use non-adapted ViDrA instead of the human ratings you collected?
We have added a description of how the rating batches were composed to appendix G:
"The crowd workers rated the images in batches of 30 images. Drawings were assigned to batches stratified by source (main study, sensitivity analysis, human reference), so each batch contained roughly the same proportions of the full set of images. This mix is also what the per-rater z-scoring relies on. Each batch was rated by 12 distinct raters. The drawings were randomly shuffled within the batch when the batch was created. All 12 raters of a batch then saw the drawings in the same order. Image filenames were hashed to not reveal the drawing's source."
An important comment about diversity in ideas across models. Our analysis is mostly focused on rating individual images. Divergent thinking, here, is used as a term to denote the ability to create something novel and effective (recognizable). Examining homogenization across independent trials was a side quest that was worth reporting in Section 4.3 as one observed failure mode, but it is not a strong focus of the paper. There has been work on idea diversity in LLMs, such as Artificial Hivemind (arXiv:2510.22954), and we will gladly acknowledge this prior work in the paper. Further, Table 2 is already quite crowded, and reporting diversity-adjusted scores would only make the table less understandable. A deeper analysis of idea diversity between models is left to future work. Same goes for within-family variability and within-model variability. We collected five images per stimulus and model, and variability is clearly visible in the snapshots on Hugging Face. Variability in outputs of tool-calling models in creative tasks over long horizons is worth investigating further in future work.
The reviewer is correct to point this out. The other sensitivity analyses draw on the collected human ratings, and so should the context ablation. We now use the human ratings in this analysis, not the ViDrA scores.