- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MasterStudyBench to evaluate the capability of models in redrawing famous paintings using code. The benchmark consists of 200 artworks, measuring basic image quality, reconstruction quality, painterly expression, and program editability. Visual evaluation employs fine-grained scoring criteria tailored to each artwork, while editability evaluation modifies the code and re-renders it to check whether changes are confined to the target region. Experiments show a weak correlation between reconstruction quality and editability.
Critical strengths
- Well-motivated separation of image quality and program editability. For image generation code, whether the output is close to the reference image and whether the program is easy to modify are distinct requirements.
- Persuasive evaluation of editability via actual code modification and re-rendering. This approach is more convincing than judging editability solely based on code style or the number of functions.
- Comprehensive validation of the evaluation methodology in the appendix. Providing repeated scoring, expert rating comparisons, image defect detection, and feedback experiments offers reasonable validation for the evaluation framework.
Critical weaknesses
-
The editability metric fails to fully verify whether the specified edits are correctly completed. Equation 8 primarily measures whether changed pixels reside within the target region, but local changes do not necessarily equate to correct edits. For instance, correctly modifying an umbrella's color versus covering the umbrella position with a solid color block could both yield high locality scores. For operations such as translation or scaling, correct results may even exceed the original region. The paper needs to clearly define the success conditions and allowable variation ranges for various operations, and explain how these checks factor into the final score.
-
Certain conclusions exceed the scope supported by the experiments. The paper attributes poor painterly expression to insufficient graphical primitives, yet it does not compare the performance of the same model using different drawing tools. The observational correlation between steps and quality is also insufficient to prove that increasing inference budget is ineffective, as difficult tasks inherently require more steps. Furthermore, Table 21 does not support that all models peak at 15–19 steps: GPT-5.6 Sol achieves a score of 69.2 at 20–29 steps, higher than 67.1 at 15–19 steps. Additionally, the correlation range in the weight analysis is 0.227–0.636, which should not be uniformly described as a stable weak correlation. Narrowing these claims to clearly distinguish experimental observations from causal speculations is recommended.
Questions for authors
See weakness.
4 comments
Response to Reviewer QZuD (PART 02)
Q3: The observational correlation between step count and quality is insufficient to show that increasing the reasoning budget is ineffective, since harder problems inherently require more steps.
R3: Thank you for this correction. We agree that the cross-sectional analysis is observational, and we do not claim that additional reasoning is ineffective in general. Our claim is more specific: for current models and within these trajectories, additional steps yield diminishing returns, and the final submission is often worse than an intermediate checkpoint (in 20–38% of trajectories). The latter is a within-trajectory comparison and is therefore not confounded by task difficulty. The revision will describe the cross-sectional step analysis as a correlation.
To further rule out difficulty as a confounder, on a new full run (Inkling-2-Small) we split the tasks into terciles by an external difficulty measure (the score of the same task under GPT-6-astra), so that difficulty is not defined by the model's own step count:
| Difficulty tercile | n | Median steps | ρ(steps, score) [95% CI] |
|---|---|---|---|
| Hard | 62 | 15 | −0.06 [−0.31, 0.20] |
| Medium | 62 | 14 | 0.19 [−0.09, 0.46] |
| Easy | 62 | 14 | 0.05 [−0.19, 0.29] |
| All (partial correlation controlling for difficulty) | 186 | 14 | 0.09 |
Key observations:
- Step count barely varies with difficulty (ρ = −0.05).
- Within each difficulty tercile, more steps do not reliably lead to higher quality (all confidence intervals contain 0). This supports the diminishing-returns conclusion without relying on the confounded cross-sectional analysis.
- To be clear, we are not saying that iterative revision via test-time scaling is useless for this task (this is exactly how human artists work). Rather, current models cannot yet carry out effective long-horizon iterative revision, and they do not exhibit the corresponding scaling behavior.
Q4: The correlation coefficients in the weighting analysis range from 0.227 to 0.636 and should not be uniformly described as consistently weak.
R4: Thank you for the correction; we fully agree. The revision will report the full range and distribution (median ρ = 0.38) and describe the result as "weak-to-moderate correlation across all weightings."
In addition, the correlation between visual quality and editability varies substantially across model–harness configurations.
We computed the correlation per configuration on the main results (one point per artwork, using a single evaluation sample per task; brackets are 95% confidence intervals from 2,000 bootstrap resamples).
| Configuration | ρ(S_base, S_edit) | ρ(S_recon, S_edit) | ρ(S_aesth, S_edit) |
|---|---|---|---|
| gpt-6-astra (Codex) | −0.03 [−0.18, 0.11] | −0.09 [−0.24, 0.06] | 0.05 [−0.09, 0.19] |
| gpt-5.6-sol (Codex) | 0.12 [−0.02, 0.26] | 0.03 [−0.11, 0.17] | 0.16 [0.02, 0.30] |
| claude-opus-5 (Claude Code) | −0.02 [−0.21, 0.16] | −0.08 [−0.27, 0.10] | −0.09 [−0.28, 0.09] |
| gemini-3.8-flash (Codex) | −0.22 [−0.36, −0.06] | −0.32 [−0.47, −0.17] | −0.04 [−0.19, 0.12] |
| doubao-seed (Codex) | 0.22 [0.06, 0.38] | 0.06 [−0.11, 0.23] | 0.30 [0.13, 0.45] |
| claude-fable-5.1 (Claude Code) | 0.59 [0.49, 0.67] | 0.41 [0.29, 0.53] | 0.61 [0.51, 0.70] |
| doubao-seed (Claude Code) | 0.60 [0.47, 0.71] | 0.31 [0.16, 0.47] | 0.54 [0.41, 0.66] |
| Inkling-2-Small (OpenCode) | 0.25 [0.12, 0.38] | 0.24 [0.11, 0.37] | 0.21 [0.07, 0.34] |
Key observations:
- In 4 configurations, visual quality and editability are essentially unrelated. For gpt-6-astra, gpt-5.6-sol, and claude-opus-5, the confidence interval of ρ(S_base, S_edit) contains 0, and gemini-3.8-flash is even negatively correlated: the better the reconstruction, the harder the program is to edit.
- Two configurations show a clear positive correlation (claude-fable-5.1 and doubao-seed under Claude Code, ρ ≈ 0.6). The newly added 8th configuration (Inkling-2-Small + OpenCode) shows a weak positive correlation (ρ = 0.25).
- The relationship varies across configurations, so visual quality cannot serve as a proxy for editability. How well a model renders does not tell us how editable its programs are; editability must be measured separately, which is exactly what this benchmark is designed to do.
Thank you for the response and additional analyses, which address most of my concerns. I will maintain my score.
Dear Reviewer,
Thank you very much for your follow-up reply. We truly appreciate your constructive guidance. We will carefully revise all discussed points based on your reviews.
Thank you again for your time and support.
Response to Reviewer QZuD (PART 01)
Thank you very much for your careful review, your positive assessment, and the concrete examples you provided. We take your concerns seriously and are glad to respond to them. Our responses are as follows.
R1: Thank you for this insightful feedback. The score does not rely on locality alone. At its core is an Agent-as-a-Judge that verifies each edit against its requirement, so the solid-block case is rejected.
R2: Thank you for the suggestion. The revision will present this explanation as a well-supported hypothesis rather than an established cause. The observation is supported by several independent signals:
Same model, different drawing methods (GPT-6-sol, same 10 paintings):