acceptodds
Under review as a conference paper at ICLR 2027

MasterStudyBench: Can Agents Paint a Painting, or Only Replay Its Pixels?

Abstract

Genuine visual intelligence transcends passive pixel imitation: true mastery requires internalizing the generative hierarchy of visual creation. In classical art, an apprentice ascends through master studies, deconstructing a masterpiece's procedural logic rather than mechanically tracing its surface. Yet current visual code synthesis remains trapped in a pixel-only regime: grading solely on rendered appearance treats code as an uninspected black box, inviting reward hacking while remaining blind to painterly aesthetics and structural modularity alike. To bridge this chasm, we introduce MasterStudyBench, evaluating procedural artistry across 200 museum masterpieces inside an offline hardened sandbox. MasterStudyBench decouples two evaluation pillars: (1) an image-conditioned rubric tree of 17,323 criteria auditing fine-grained aesthetics across brushwork, lighting, and composition; and (2) an executed editability probe measuring re-rendered spatial deltas under counterfactual source mutations to test program modularity. Across frontier multimodal models, a profound disconnect emerges: adversarial stubs capturing nearly 67% pixel rewards collapse to 0.0 under MasterStudyBench, while rendered visual fidelity correlates only weakly with executed editability (Spearman's ), proving that visually convincing paintings routinely conceal uneditable spaghetti code. Mechanistically, architectural planning consistently yields a 2.6-4.7% editability gain by enforcing scene decomposition, while painterly aesthetics lags 11-16% behind basic visual integrity, exposing procedural shaders as the critical capability frontier. By separating painting from replaying, MasterStudyBench charts the path from superficial mimicry toward authentic procedural intelligence.

Primary area:
datasets and benchmarks
Keywords:
Painting, Drawing, Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QZuDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MasterStudyBench to evaluate the capability of models in redrawing famous paintings using code. The benchmark consists of 200 artworks, measuring basic image quality, reconstruction quality, painterly expression, and program editability. Visual evaluation employs fine-grained scoring criteria tailored to each artwork, while editability evaluation modifies the code and re-renders it to check whether changes are confined to the target region. Experiments show a weak correlation between reconstruction quality and editability.

Critical strengths
  • Well-motivated separation of image quality and program editability. For image generation code, whether the output is close to the reference image and whether the program is easy to modify are distinct requirements.
  • Persuasive evaluation of editability via actual code modification and re-rendering. This approach is more convincing than judging editability solely based on code style or the number of functions.
  • Comprehensive validation of the evaluation methodology in the appendix. Providing repeated scoring, expert rating comparisons, image defect detection, and feedback experiments offers reasonable validation for the evaluation framework.
Critical weaknesses
  • The editability metric fails to fully verify whether the specified edits are correctly completed. Equation 8 primarily measures whether changed pixels reside within the target region, but local changes do not necessarily equate to correct edits. For instance, correctly modifying an umbrella's color versus covering the umbrella position with a solid color block could both yield high locality scores. For operations such as translation or scaling, correct results may even exceed the original region. The paper needs to clearly define the success conditions and allowable variation ranges for various operations, and explain how these checks factor into the final score.

  • Certain conclusions exceed the scope supported by the experiments. The paper attributes poor painterly expression to insufficient graphical primitives, yet it does not compare the performance of the same model using different drawing tools. The observational correlation between steps and quality is also insufficient to prove that increasing inference budget is ineffective, as difficult tasks inherently require more steps. Furthermore, Table 21 does not support that all models peak at 15–19 steps: GPT-5.6 Sol achieves a score of 69.2 at 20–29 steps, higher than 67.1 at 15–19 steps. Additionally, the correlation range in the weight analysis is 0.227–0.636, which should not be uniformly described as a stable weak correlation. Narrowing these claims to clearly distinguish experimental observations from causal speculations is recommended.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QZuD (PART 01)

Thank you very much for your careful review, your positive assessment, and the concrete examples you provided. We take your concerns seriously and are glad to respond to them. Our responses are as follows.

*Q1: The Editability metric does not sufficiently verify that edits are performed correctly (e.g., correctly recoloring an umbrella vs. covering it with a solid block; the correct result of a translation or scaling may extend beyond the original region). Please define success conditions and tolerances for each type of operation and explain how these checks enter the final score.

R1: Thank you for this insightful feedback. The score does not rely on locality alone. At its core is an Agent-as-a-Judge that verifies each edit against its requirement, so the solid-block case is rejected.

  1. Covering the umbrella with a solid block → rejected. Such an edit hits the target region but fails the "was the requested effect achieved" check: the umbrella is erased rather than recolored, which the judge can see directly by comparing the pre- and post-edit renders.
  2. We tested this scenario directly. Taking 50 real edits that the verifier had originally accepted, we replaced each edited region with a single solid block and re-judged the results with the unchanged verification judge of the released verifier: 44/50 (88%) were rejected.
  3. The same mechanism also catches real "local but wrong" edits in our data (see our R1 to Reviewer aAvj): a recolor that changes only one of the façade's several polygons and covers 1.0% of the canvas (capped at 2), and a pavilion removal that covers 4.1% of the canvas but disturbs the surrounding trees (capped at 1).
  4. Translation and scaling: these are judged by whether the element was moved or scaled as requested, not by whether the change falls within the original region.
  5. Final score: each leaf receives a 0–4 score from the judge, subject to the hard cap (no change / fails to execute / breaks the render ≤ 1). The revision will add the success conditions and tolerances for each operator, together with example images of passing and failing probes.

Q2: The paper attributes the limited painterly expressiveness to insufficient graphics primitives, but does not compare the same model using different drawing tools.

R2: Thank you for the suggestion. The revision will present this explanation as a well-supported hypothesis rather than an established cause. The observation is supported by several independent signals:

  • Aesthetics is the lowest-scoring visual dimension in every configuration, medium, and genre (1.66–1.83 on a 0–4 scale).
  • Texture-related primitives (compositing modes, filters) are used in fewer than 30% of submissions in every group. In the full Inkling-2-Small / gpt-6-sol runs, compositing modes, filters, or patterns appear in only 4–10% of submissions.
  • Going further, we had the agent paint directly with brushes, exposing the full GIMP brush set, yet the models found it even harder to paint well: on the same 10 paintings, GIMP brushes obtain a total visual score of 65.9, 6–7 points lower than Canvas 2D (71.6) and p5.js (72.6) (a relative drop of 8–9%), while editability falls from 62–67 to 40, a drop of more than 20 points.
  • We also re-ran the Canvas 2D version of these 10 paintings with the same model (GPT-6-sol, Codex harness). On these paintings, Canvas 2D obtains a total visual score of 76.1 vs. 65.2 for GIMP brushes on the same paintings, with the program scoring higher on all paintings (mean +10.9 points, Wilcoxon p < 0.01); editability is 69.7 vs. 45.5.

Same model, different drawing methods (GPT-6-sol, same 10 paintings):

Drawing methodTasksBasicRecon.AestheticEditability
Painting2Code (Canvas 2D program)1083.672.163.266.5
Painting2Code (p5.js program)1084.271.368.962.1
Painting2Brush (GIMP brushes)1075.164.663.640.2
AuthorsOpenReview ↗

Response to Reviewer QZuD (PART 02)

Q3: The observational correlation between step count and quality is insufficient to show that increasing the reasoning budget is ineffective, since harder problems inherently require more steps.

R3: Thank you for this correction. We agree that the cross-sectional analysis is observational, and we do not claim that additional reasoning is ineffective in general. Our claim is more specific: for current models and within these trajectories, additional steps yield diminishing returns, and the final submission is often worse than an intermediate checkpoint (in 20–38% of trajectories). The latter is a within-trajectory comparison and is therefore not confounded by task difficulty. The revision will describe the cross-sectional step analysis as a correlation.

To further rule out difficulty as a confounder, on a new full run (Inkling-2-Small) we split the tasks into terciles by an external difficulty measure (the score of the same task under GPT-6-astra), so that difficulty is not defined by the model's own step count:

Difficulty tercilenMedian stepsρ(steps, score) [95% CI]
Hard6215−0.06 [−0.31, 0.20]
Medium62140.19 [−0.09, 0.46]
Easy62140.05 [−0.19, 0.29]
All (partial correlation controlling for difficulty)186140.09

Key observations:

  • Step count barely varies with difficulty (ρ = −0.05).
  • Within each difficulty tercile, more steps do not reliably lead to higher quality (all confidence intervals contain 0). This supports the diminishing-returns conclusion without relying on the confounded cross-sectional analysis.
  • To be clear, we are not saying that iterative revision via test-time scaling is useless for this task (this is exactly how human artists work). Rather, current models cannot yet carry out effective long-horizon iterative revision, and they do not exhibit the corresponding scaling behavior.

Q4: The correlation coefficients in the weighting analysis range from 0.227 to 0.636 and should not be uniformly described as consistently weak.

R4: Thank you for the correction; we fully agree. The revision will report the full range and distribution (median ρ = 0.38) and describe the result as "weak-to-moderate correlation across all weightings."

In addition, the correlation between visual quality and editability varies substantially across model–harness configurations.

We computed the correlation per configuration on the main results (one point per artwork, using a single evaluation sample per task; brackets are 95% confidence intervals from 2,000 bootstrap resamples).

Configurationρ(S_base, S_edit)ρ(S_recon, S_edit)ρ(S_aesth, S_edit)
gpt-6-astra (Codex)−0.03 [−0.18, 0.11]−0.09 [−0.24, 0.06]0.05 [−0.09, 0.19]
gpt-5.6-sol (Codex)0.12 [−0.02, 0.26]0.03 [−0.11, 0.17]0.16 [0.02, 0.30]
claude-opus-5 (Claude Code)−0.02 [−0.21, 0.16]−0.08 [−0.27, 0.10]−0.09 [−0.28, 0.09]
gemini-3.8-flash (Codex)−0.22 [−0.36, −0.06]−0.32 [−0.47, −0.17]−0.04 [−0.19, 0.12]
doubao-seed (Codex)0.22 [0.06, 0.38]0.06 [−0.11, 0.23]0.30 [0.13, 0.45]
claude-fable-5.1 (Claude Code)0.59 [0.49, 0.67]0.41 [0.29, 0.53]0.61 [0.51, 0.70]
doubao-seed (Claude Code)0.60 [0.47, 0.71]0.31 [0.16, 0.47]0.54 [0.41, 0.66]
Inkling-2-Small (OpenCode)0.25 [0.12, 0.38]0.24 [0.11, 0.37]0.21 [0.07, 0.34]

Key observations:

  • In 4 configurations, visual quality and editability are essentially unrelated. For gpt-6-astra, gpt-5.6-sol, and claude-opus-5, the confidence interval of ρ(S_base, S_edit) contains 0, and gemini-3.8-flash is even negatively correlated: the better the reconstruction, the harder the program is to edit.
  • Two configurations show a clear positive correlation (claude-fable-5.1 and doubao-seed under Claude Code, ρ ≈ 0.6). The newly added 8th configuration (Inkling-2-Small + OpenCode) shows a weak positive correlation (ρ = 0.25).
  • The relationship varies across configurations, so visual quality cannot serve as a proxy for editability. How well a model renders does not tell us how editable its programs are; editability must be measured separately, which is exactly what this benchmark is designed to do.

Thank you for the response and additional analyses, which address most of my concerns. I will maintain my score.

Dear Reviewer,

Thank you very much for your follow-up reply. We truly appreciate your constructive guidance. We will carefully revise all discussed points based on your reviews.

Thank you again for your time and support.

Reviewer_VZXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MasterStudyBench, a benchmark for reconstructing 200 museum paintings as executable drawing programs. It evaluates rendered outputs using image-conditioned rubrics and measures program editability through source-code mutations and subsequent rendering changes. The evaluation covers several model–harness configurations, alongside studies of judge reliability, expert agreement, and iterative refinement. The main reported finding is that reconstruction quality is only weakly correlated with editability, suggesting that visual resemblance and program controllability should be evaluated separately.

Critical strengths
  1. The paper addresses a useful evaluation problem. A convincing rendering does not necessarily imply that the underlying program can be meaningfully edited. Evaluating programs through actual modifications is a worthwhile direction for assessing visual code generation.

  2. The benchmark provides substantial domain coverage. The 200 artworks span different subjects, media, and artistic traditions. The 17,323 image-conditioned criteria offer a detailed basis for assessing aspects such as composition, brushwork, and illumination that a single similarity score may overlook.

  3. The authors make a meaningful effort to validate the evaluation. The paper includes repeated scoring, controlled image corruptions, comparisons with perceptual metrics, and expert ratings. These experiments provide useful initial evidence for the rubric-based approach beyond simply presenting model rankings.

Critical weaknesses
  1. Does the editability score measure successful semantic editing, or mainly the locality of pixel changes? Equation 8 rewards changes contained within an entity’s footprint, but a small, ineffective modification could satisfy this criterion without completing the requested edit. Conversely, a correct translation or enlargement may legitimately change pixels outside the original footprint. Could the authors explain how semantic success is verified, how these geometric edits are handled, and how probe outcomes are converted into the scores used in Equation 5?

  2. How much of the rubric evaluator’s advantage comes from hierarchy rather than additional evaluation effort? Appendix A.2.1 uses approximately 22 judge calls per case for the proposed method, compared with one call for flat and holistic scoring. This makes it difficult to attribute the gains in stability and human agreement specifically to hierarchical structure. Could the authors include a budget-matched flat atomic checklist and an ensemble of holistic judgments?

  3. Could the authors reconcile the experimental populations and numerical summaries across the paper? Table 3 lists 45 narrative artworks and 54 landscapes, whereas Table 20 reports 102 and 31, respectively, while still totaling 200 cases. In Table 12, the operator-level no-op rates imply approximately 10/104, or 9.6%, rather than the reported aggregate of 5.8%. Is there anything I miss here?

BTW, I don’t seem to be able to find the "AI Use Statement" required by this year’s submission guidelines.

Questions for authors

My main questions are included in the three concerns under Critical Weaknesses. I would be open to raising my score if the authors can satisfactorily address most of these concerns.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VZXR (PART 01)

Thank you very much for your careful review and for clearly stating the issues you care about most. We take them seriously and respond to each of them below.

Q1: Does the Editability score measure semantic edit success, or mainly the locality of pixel changes? How is semantic success verified? How are geometric edits such as translation and scaling handled? How are probe results converted into the score in Eq. 5?

R1: Thank you for this insightful question. The score measures semantic success; Eq. 8 is only a diagnostic quantity that the judge consults, not the final score. The final verdict on semantic correctness is made entirely by the multimodal Agent-as-a-Judge, and the pixel differences in Eq. 7–8 serve only as auxiliary diagnostic signals and visual cues; they do not directly produce the score.

  1. What the judge sees: the pre-edit render, the post-edit render, a difference overlay, and the modified source code.
  2. What the judge checks: (i) whether the referenced element was modified; (ii) whether the requested effect was achieved; (iii) whether protected elements remain unchanged.
  3. Hard cap: edits that make no change, fail to execute, or break the render receive at most 1 point.

Below are two real cases from the benchmark that are local according to Eq. 8 but are rejected by the judge:

CaseEditΔ (Eq. 8)What actually happenedJudge verdict
met-49072Remove the pavilion at the bottom right (comment out pavilion(168,404,1);)4.1% (≪ 0.25)The pavilion is removed, but the entire grove of trees is redrawn and the inscription changes: the function consumes seeded random numbers, so removing it shifts every subsequent strokeProtected region not preserved → capped at 1
aic-20684Recolor the right façade #895c59 → #00d9ff1.0%The change is clean and local, but only a narrow strip turns cyan, because the façade is drawn as several separate polygonsEffect only partially achieved → capped at 2
  • Translation and scaling: these are judged by whether the element was moved or scaled as requested and whether the rest of the content remains unchanged, not by whether the change falls within the original region.
  • Conversion to Eq. 5: each Editability leaf receives a 0–4 score from the judge, subject to the hard cap (no change / fails to execute / breaks the render ≤ 1). S_edit is the weighted average over all Editability leaves.

Q2: How much of the rubric's advantage comes from its hierarchical structure, and how much from greater evaluation effort (~22 judge calls vs. 1)? Could you add a budget-matched flat atomic checklist and an ensemble of multiple holistic judgments?

R2: Thank you for the suggestion. We added experiments following exactly this design: flat and holistic scoring were each repeated k = 1, 5, 10, and 22 times and averaged, and compared on the 30 expert-scored artworks. At the same budget of 22 calls, the rubric still leads by a large margin.

ProtocolJudge calls / itemρ overallρ fidelityρ qualityStd. of item score
Hierarchical rubric (single pass)~220.93 [0.91, 0.96]0.920.93—
Flat, k = 110.71 [0.64, 0.76]0.780.783.55
Flat, k = 550.72 [0.69, 0.75]0.770.791.59
Flat, k = 10100.73 [0.70, 0.74]0.780.791.12
Flat, k = 22220.73 [0.69, 0.74]0.770.790.76
Holistic, k = 110.61 [0.51, 0.70]0.630.674.32
Holistic, k = 550.63 [0.60, 0.67]0.710.741.93
Holistic, k = 10100.63 [0.61, 0.65]0.690.721.37
Holistic, k = 22220.63 [0.60, 0.65]0.690.710.92

Key observations:

  • The gap does not shrink as the budget grows: 0.93 vs. 0.73 (flat) and 0.63 (holistic).
  • Extra calls only reduce noise: the baselines' standard deviation drops by roughly 4–5×, but their agreement with experts stops improving after k = 5–10.
  • The advantage comes from per-criterion decomposition, not from evaluation compute.
AuthorsOpenReview ↗

Response to Reviewer VZXR (PART 02)

Q3: Could you double-check the sample and count summaries across experiments? Table 3 lists 45 narrative and 54 landscape paintings, whereas Table 20 lists 102 and 31.

R3: Thank you for the careful check. Both tables cover the same 200 artworks; the genre counts differ because the two tables use different label sources.

The discrepancy between Table 3 and Table 20 stems mainly from different label sources. Table 3 uses the genre labels assigned at curation time. For Table 20, we manually re-verified the genre of every task, took medium labels from each museum's collection API, and, during re-verification, merged figure scenes and narrative scenes into a single "figure / narrative" class (102 artworks), with portraits listed separately.

We agree that two taxonomies in one paper are confusing; the revision will state the label source in each table caption and explain the mapping between the two label sets in the appendix (the curation labels describe the corpus, while the finer, manually verified labels are used for per-model analysis).

Q4: Based on the per-operator no-op rates in Table 12, the aggregate should be 10/104 = 9.6%, not the reported 5.8%.

R4: Thank you for the careful check; you are correct. Table 12 in the submission reports operator-level statistics sampled from 11 artworks. The per-operator rows are correct, but the no-op ratio in the aggregate row was miscalculated: it should be 9.6% (10/104) rather than 5.8%.

To avoid such calculation errors and misreadings caused by a small sample, we have replaced the sampled table with statistics over all 9,375 executed probes: the no-op rate is 8.7% and the bleed rate is 2.6%.

Operator familynMedian ΔMean ΔBleedNo-op
Spatial translation2,7260.00490.01940.8%7.8%
Extent rescaling160.01620.01490.0%12.5%
Element visibility8370.00840.03221.8%11.4%
Loop-bound & other2490.00860.03832.0%12.4%
Colour and stroke3,1850.00930.04764.6%10.0%
Compound & structural2,3620.00970.03662.2%6.5%
All realized probes9,3750.00760.03502.6%8.7%

Q5: The "AI Use Statement" required by this year's submission guidelines seems to be missing.

R5: Thank you for the reminder. The statement is provided in Appendix A.6 ("AI Assistance Statement"). In the revision, we will move it to the designated position after the main text, as required by the formatting guidelines.

We hope that these clarifications and the new full-scale experiments address your main concerns, and we would be happy to answer any further questions.

If our response can address your concerns, we hope you can RECONSIDER our score.

Dear Reviewer VZXR,

Thank you for your review. Since the discussion time is less than a day, we would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Reviewer_aAvjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MasterStudyBench evaluates agents that reconstruct paintings as executable drawing programs rather than directly returning raster images. It combines 200 museum artworks with hierarchical visual rubrics, painterly-aesthetic criteria, execution-based editability probes, and a sandbox that rejects prohibited shortcuts. The experiments compare model-harness configurations and examine human alignment, perturbation sensitivity, and refinement trajectories. The central finding is that visual reconstruction quality and the editability of the generated program are not interchangeable.

Critical strengths

(1) I think the problem this paper investigate is a "real" question that the researchers in this field meet and this is an important question. (2) Separating rendered appearance from executable program behavior is a meaningful evaluation contribution. An agent can produce a visually plausible reconstruction without exposing a program that supports useful edits, and the benchmark attempts to measure this distinction rather than relying exclusively on perceptual similarity. (3) The paper provides substantial rubric construction and several validation studies, including expert audits, controlled visual perturbations, and comparisons with alternative scoring protocols. These analyses support the usefulness of finer-grained evaluation and help reveal limitations that a single image-similarity score would miss.

Critical weaknesses

(1) The validity and coverage of the editability score are not sufficiently established. Equations 7 and 8 primarily measure the spatial containment of a luminance difference mask, which does not by itself verify that the intended semantic edit succeeded or that the resulting image remains coherent. The metric may also depend on which targets the mutation mechanism can reach. Table 12 reports only 104 realized probes, while the benchmark contains 200 artworks and the main study evaluates multiple configurations, leaving unclear how representative these probes are and how unavailable targets enter the final score. (2) Several conclusions are stronger than the supporting comparisons. The hierarchical protocol uses approximately 22 judge calls per case, whereas flat and holistic alternatives use substantially fewer, so its advantage is not cleanly separated from additional evaluation compute. (3) Likewise, weak correlations and small trajectory studies support a distinction between the measured scores but do not establish an intrinsic separation between visual quality and generative mastery. The sensitivity analysis includes correlations as high as 0.636, which also warrants a more qualified interpretation.

Questions for authors

(1) How is a probe checked for successful execution of the intended semantic change, beyond the locality of the changed pixels? Please show representative cases where a localized change is nevertheless incorrect, explain the treatment of luminance-preserving recoloring, and describe how unreachable or occluded targets affect the score. (2) What population is represented by the 104 realized probes in Table 12, and how does it relate to the 150 masks in the expert audit and the full evaluation? Please report probe coverage by artwork and model configuration, including planned, executable, failed, and unreachable probes, with the aggregation denominator. (3) Does the hierarchical rubric retain its human-alignment advantage against flat or holistic scoring under matched judge-call or token budgets? A repeated flat-criteria baseline using the same criteria and comparable compute would help isolate the contribution of hierarchy. (4) The relevant works about evaluating visual intelligence beyond pixels could be included into related work discussion: JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent; IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering; JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents (5) Are the correlations between visual quality and editability stable within individual model-harness configurations, rather than only in pooled results? Please provide uncertainty estimates and clarify the sample sizes and controlled conditions behind the claimed planning-related editability gains.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aAvj (PART 01)

Thank you for your positive assessment and your detailed questions. We take your concerns seriously and are glad to address them. Our responses are as follows.

Q1: Beyond the locality of changed pixels, how do you verify that an editability probe actually accomplished the intended semantic change? Please show cases where the change is local but wrong, explain how luminance-preserving recolors are handled, and describe how unreachable or occluded targets affect the score.

R1: Thank you for this insightful question. We first clarify our core evaluation mechanism: the final verdict on semantic correctness is made entirely by a multimodal Agent-as-a-Judge. The pixel differences in Eq. 7–8 serve only as auxiliary diagnostic signals and visual cues; they do not directly produce the score.

  1. What the judge sees: the pre-edit render, the post-edit render, a difference overlay, and the modified source code.
  2. What the judge checks: (i) whether the referenced element was modified; (ii) whether the requested effect was achieved; (iii) whether protected elements remain unchanged.
  3. Hard cap: edits that make no change, fail to execute, or break the render receive at most 1 point.

Below are two real cases from the benchmark that are local according to Eq. 8 but are rejected by the judge:

CaseEditΔ (Eq. 8)What actually happenedJudge verdict
met-49072Remove the pavilion at the bottom right (comment out pavilion(168,404,1);)4.1% (≪ 0.25)The pavilion is removed, but the entire grove of trees is redrawn and the inscription changes: the function consumes seeded random numbers, so removing it shifts every subsequent strokeProtected region not preserved → capped at 1
aic-20684Recolor the right façade #895c59 → #00d9ff1.0%The change is clean and local, but only a narrow strip turns cyan, because the façade is drawn as several separate polygonsEffect only partially achieved → capped at 2

Q2: What sample do the 104 realized probes in Table 12 represent? How do they relate to the 150 masks in the expert audit and to the full evaluation? Please report probe coverage and aggregate denominators per artwork and per configuration.

R2: Thank you for this careful question. Table 12 in the submission is a sampled statistic (11 artworks), and the 150 masks form a separate, independent sample used for human verification of the difference masks. Following your suggestion, we have recomputed Table 12 over all 9,375 executed probes in the full evaluation.

Table A1 (replaces Table 12): All executed probes. Δ is the fraction of canvas pixels that changed; bleed counts probes exceeding the 0.25 containment threshold; no-op counts probes with an empty mask.

Operator familynMedian ΔMean ΔBleedNo-op
Spatial translation2,7260.00490.01940.8%7.8%
Extent rescaling160.01620.01490.0%12.5%
Element visibility8370.00840.03221.8%11.4%
Loop-bound & other2490.00860.03832.0%12.4%
Colour and stroke3,1850.00930.04764.6%10.0%
Compound & structural2,3620.00970.03662.2%6.5%
All realized probes9,3750.00760.03502.6%8.7%

Key observations:

  • Edits are highly local on the full sample. The median Δ is 0.0076 and only 2.6% of probes bleed, mostly from colour and stroke edits (4.6%), consistent with the conclusion of the sampled table.
  • Execution failures are rare. Only 0.2% of probes actually fail to execute.
  • No leaf is silently dropped. Leaves not executed due to budget and leaves with no executable probe remain in the Eq. 5 denominator and receive at most 1 point.

Table A2: Probe coverage per configuration. The Eq. 5 denominator consists of all Editability leaves of the configuration.

ConfigurationEditability leaves (denominator)PlannedExecutedFailedNot executed (budget)No executable probe
gpt-6-astra (Codex)2,5381,985 (78.2%)1,740 (68.6%)6 (0.2%)239 (9.4%)539 (21.2%)
gpt-5.6-sol (Codex)2,5382,087 (82.2%)1,821 (71.7%)4 (0.2%)262 (10.3%)438 (17.3%)
claude-fable-5.1 (Claude Code)2,5381,897 (74.7%)1,682 (66.3%)5 (0.2%)210 (8.3%)604 (23.8%)
AuthorsOpenReview ↗

Response to Reviewer aAvj (PART 02)

Q3: When the number of judge calls or the token budget is matched, does the hierarchical rubric's advantage in human agreement over flat or holistic scoring still hold?

R3: Thank you for raising this key question. The answer is yes. Following your suggestion, on the 30 expert-scored artworks from Exp. 1.1, we repeated flat and holistic scoring k = 1, 5, 10, and 22 times and averaged the results, where k = 22 matches the ~22 judge calls of a single hierarchical-rubric pass. Brackets denote bootstrap 95% confidence intervals.

ProtocolJudge calls / itemρ overallρ fidelityρ qualityStd. of item score
Hierarchical rubric (single pass)~220.93 [0.91, 0.96]0.920.93—
Flat, k = 110.71 [0.64, 0.76]0.780.783.55
Flat, k = 550.72 [0.69, 0.75]0.770.791.59
Flat, k = 10100.73 [0.70, 0.74]0.780.791.12
Flat, k = 22220.73 [0.69, 0.74]0.770.790.76
Holistic, k = 110.61 [0.51, 0.70]0.630.674.32
Holistic, k = 550.63 [0.60, 0.67]0.710.741.93
Holistic, k = 10100.63 [0.61, 0.65]0.690.721.37
Holistic, k = 22220.63 [0.60, 0.65]0.690.710.92

Key observations:

  • More calls bring stability, not agreement with experts. As k increases, the per-item score standard deviation drops from 3.55 to 0.76 (flat) and from 4.32 to 0.92 (holistic), but agreement with experts already saturates at k = 5–10.
  • The gap remains large at a matched budget. At k = 22, flat scoring reaches 0.73 and holistic scoring 0.63, far below the 0.93 of a single rubric pass.
  • The advantage comes from per-criterion decomposition, not from evaluation compute. Averaging only removes the judge's noise on criterion-free scores; it cannot recover the information provided by decomposing the assessment into individual criteria.

Q4: The related work could include evaluations of visual intelligence "beyond pixels": JarvisArt, IR3D-Bench, and JarvisHub.

R4: Thank you for these references. We will discuss all three works in Related Work. JarvisArt (an intelligent photo-retouching agent) and JarvisHub (an open framework for canvas-oriented creative agents) evaluate agents that operate creative tools, while IR3D-Bench evaluates scene understanding through agentic inverse rendering. MasterStudyBench differs from them in that the executable program itself is the deliverable, and we test it by mutating the program. We therefore measure whether the artifact can be edited, not merely whether the output looks correct.

AuthorsOpenReview ↗

Response to Reviewer aAvj (PART 03)

Q5: Is the correlation between visual quality and editability stable within individual model–harness configurations? Please provide uncertainty estimates.

R5: Thank you for this careful and constructive suggestion.

First, we would like to clarify ρ = 0.636: this is the maximum over the 231 weightings of the visual composite score, not a typical value (the median is 0.38). Even at this maximum, the editability ranking agrees only partially with visual quality. The revision will report the full distribution and describe the result as "weak-to-moderate correlation across all weightings."

Second, the correlation between visual quality and editability varies substantially across model–harness configurations. We computed the correlation per configuration on the main results (one point per artwork, using a single evaluation sample per task; brackets are 95% confidence intervals from 2,000 bootstrap resamples).

Configurationρ(S_base, S_edit)ρ(S_recon, S_edit)ρ(S_aesth, S_edit)
gpt-6-astra (Codex)−0.03 [−0.18, 0.11]−0.09 [−0.24, 0.06]0.05 [−0.09, 0.19]
gpt-5.6-sol (Codex)0.12 [−0.02, 0.26]0.03 [−0.11, 0.17]0.16 [0.02, 0.30]
claude-opus-5 (Claude Code)−0.02 [−0.21, 0.16]−0.08 [−0.27, 0.10]−0.09 [−0.28, 0.09]
gemini-3.8-flash (Codex)−0.22 [−0.36, −0.06]−0.32 [−0.47, −0.17]−0.04 [−0.19, 0.12]
doubao-seed (Codex)0.22 [0.06, 0.38]0.06 [−0.11, 0.23]0.30 [0.13, 0.45]
claude-fable-5.1 (Claude Code)0.59 [0.49, 0.67]0.41 [0.29, 0.53]0.61 [0.51, 0.70]
doubao-seed (Claude Code)0.60 [0.47, 0.71]0.31 [0.16, 0.47]0.54 [0.41, 0.66]
Inkling-2-Small (OpenCode)0.25 [0.12, 0.38]0.24 [0.11, 0.37]0.21 [0.07, 0.34]

Key observations:

  • In 4 configurations, visual quality and editability are essentially unrelated. For gpt-6-astra, gpt-5.6-sol, and claude-opus-5, the confidence interval of ρ(S_base, S_edit) contains 0, and gemini-3.8-flash is even negatively correlated: the better the reconstruction, the harder the program is to edit.
  • Two configurations show a clear positive correlation (claude-fable-5.1 and doubao-seed under Claude Code, ρ ≈ 0.6). The newly added 8th configuration (Inkling-2-Small + OpenCode) shows a weak positive correlation (ρ = 0.25).
  • The relationship varies across configurations, so visual quality cannot serve as a proxy for editability. How well a model renders does not tell us how editable its programs are; editability must be measured separately, which is exactly what this benchmark is designed to do.

Q6: Please describe the sample size and controlled conditions behind the planning-related editability gains.

R6: Regarding the planning-related gains in Appendix A.4: the comparison fixes the set of models and the tasks and varies only the instruction (architectural planning with explicit entity decomposition vs. reactive trial-and-error patching). The sample is 6 models × 200 bench tasks each × 3 evaluations per task per paradigm (the same runs used for the state-machine table in Appendix A.3).

Dear Reviewer aAvj,

Thank you for your review. Since the discussion time is less than a day, we would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.