acceptodds
Under review as a conference paper at ICLR 2027

SolveEdit: Benchmarking Visual Problem Solving in Generative Models

Abstract

Machine intelligence is often evaluated through abstract reasoning problems, yet many real-world problems are visual, such as arranging objects, repairing layouts, or tracing routes. Solving these problems requires understanding a scene, inferring what must change to achieve a goal, and realizing that change without disturbing unrelated content. However, existing benchmarks mainly evaluate perception, generation, or explicitly specified transformations, leaving goal-driven visual problem solving underexplored. To bridge this gap, we introduce \solveedit, a benchmark for visual problem solving through scene transformation. Given an image and a goal, a model must infer a valid transformation from the request, the scene, or a visually expressed rule, then execute it while preserving unrelated content. \solveedit contains 2,728 cases across 10 domains and 54 subdomains. Atomic transition contracts specify required and protected conditions, enabling \solvescore to measure completion and unintended changes without a single reference output. The strongest evaluated model achieves only 57.0% \solvescore. We further introduce \solveeditplan, a two-stage visual planner that instantiates the transition before generation. Under matched single-generation evaluation, it improves \solvescore by 9.1 points on average across three tested generators, including a gain from 57.0% to 71.6% for GPT-Image-2, without modifying the editor.

Primary area:
datasets and benchmarks
Keywords:
benchmark, image editing, visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dGtrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SolveEdit, a benchmark for visual problem solving through scene transformation. As authors claimed, existing image-editing benchmarks evaluate whether a model can execute an explicitly specified change. The proposed SolveEdit instead asks the model to infer what change is needed from the source image and a high-level goal. The benchmark contains 2,728 cases across 10 domains and 54 subdomains, organized by the information source that determines the required transition: Instruction-Specified (IS, 736 cases), State-Dependent (SD, 629 cases), and Rule-Dependent (RD, 1,363 cases). Each case has an atomic transition contract with required conditions and protected conditions. SolveScore is used to measure completion minus a bounded penalty for collateral damage. Authors evaluate 11 models (9 image-to-image, 2 image-to-video) and find that the strongest model achieves only 57.0% SolveScore. They further introduce SolveEdit-Plan, a two-stage test-time planner that instantiates the transition before a single call to an unchanged editor. Under matched budget, SolveEdit-Plan improves GPT-Image-2 from 57.0% to 71.6%, outperforming a call-matched generic rewrite control by 6.3 points, with larger gains on SD (+23.3%) and RD (+38.0%) than on IS (+12.0%).

Critical strengths
  1. This paper has a clean problem formulation and orthogonal evaluation axis. The IS/SD/RD decomposition separates instruction following from scene understanding and rule inference within a single generative task. This is orthogonal to content domain and task type. The same spatial operation can land in any regime depending on what fixes the valid outcome. This axis has not been systematically isolated in prior editing benchmarks.
  2. The contract-based scoring avoids reference-image bias. By specifying required and protected atomic conditions rather than matching a single reference output, SolveScore accommodates multiple valid solutions and separately measures completion and collateral damage. The six diagnostic cells provide interpretable failure attribution.
Critical weaknesses
  1. The planner generalization is untested. The planning experiments use only GPT-5.6 Sol as the planner, and the 14.6% gain may largely reflect this model's strong visual reasoning ability rather than the planning procedure itself. No other planner backbone is tested. It is unclear whether smaller or lightweight planner models can achieve similar gains, or whether the procedure transfers at all. The 0% planner failure rate reported by the authors likely reflects GPT-5.6 Sol's capability rather than task ease. Additionally, since both the planner and the evaluator use GPT-5.6 Sol, the absolute SolveScore improvement may carry some systematic alignment between plan and judge, though this does not affect the model ranking under direct evaluation.
  2. The benchmark size and distribution looks somewhat imbalanced. With 2,728 cases, the benchmark is modest in scale. RD accounts for 1,363 cases while IS has only 736. The per-subdomain average is about 50 cases, which limits statistical power for fine-grained claims. While all evaluated outputs use 1024×1024 resolution, the coverage of different aspect ratios is not discussed.
Questions for authors
  1. SolveEdit-Plan uses only GPT-5.6 Sol as the planner. Have authors ever tested the planning procedure with a smaller or lightweight planner model on a subset? Do the gains persist, or do they largely reflect GPT-5.6 Sol's raw visual reasoning ability?
  2. Table 8 shows RA human agreement is only 91.8% (vs. 96.6% SA, 97.3% VQ). Since the paper's main diagnostic claim is that relational conditions are the dominant failure mode, how sensitive are the regime gaps (Figure 5b) and the model rankings to RA scoring noise? Have authors re-estimated the main findings after excluding criteria where the evaluator abstains or where human agreement is lowest?
  3. The RD accounts for 50% of the benchmark. What types of in-image rules are most common (legends, constraints, repeated patterns, game rules)? Are the RD cases solvable by a strong text-only LLM without the image, or do they genuinely require visual rule reading?
  4. Both SolveEdit-Plan and SolveScore use GPT-5.6 Sol. Does this create a systematic alignment between what the planner produces and what the evaluator rewards? Have authors compared direct model rankings or planning gains when evaluated with a different VLM judge?
5 comments
AuthorsOpenReview ↗

Part 1/3: Planner Backbones (Q1) and Planner–Judge Alignment (Q4)

We thank the reviewer for the positive assessment and the precise questions. Unless stated otherwise, every experiment below runs on the full 2,728-case benchmark with GPT-Image-2 as the fixed editor (one generation per case) and the same evaluation prompt as Section 5.4.

Q1. Do the gains persist with other planners?

Yes. We reran the planning comparison with three additional planners: two closed models from other vendors and one open-weight model. To separate the procedure from the backbone's raw ability, each planner is also run as a Generic Vision Rewrite (GVR) on the same backbone.

PlannerGVRSolveEdit-PlanPlan − DirectPlan − GVR (same backbone)
GPT-5.6 Sol (paper)65.371.6+14.6+6.3
Gemini 3.1 Pro64.870.9+13.9+6.1
Claude Sonnet 5.568.772.8+15.8+4.1
Qwen3.5-27B (open-weight)60.563.8+6.8+3.3

Direct is 57.0 for all rows. SolveEdit-Plan beats the same-backbone GVR for every planner, so the procedure adds value beyond what each backbone achieves by rewriting the prompt. The reviewer is also right that planner capability matters: with Qwen3.5-27B the gain over Direct is about half that of the closed planners. This is consistent with the benchmark's diagnosis, since the procedure helps to the extent that the planner can determine the transition from the image. We have not tested planners smaller than 27B. We agree that the 0% planner failure rate reflects GPT-5.6 Sol's capability rather than task ease, and have rephrased it in Appendix E. These results are in Appendix F.7.

Q4. Does using GPT-5.6 Sol as both planner and judge create alignment?

The design limits the channel for alignment. The planner never sees the contract, and the judge never sees the planner's text: it scores the generated image against the source image, the request and the contract. To test for residual alignment, we re-scored the same images with judges from two other vendors.

Planning gains.

JudgeDirectGVRSolveEdit-PlanPlan − DirectPlan − GVR
GPT-5.6 Sol (paper)57.065.371.6+14.6+6.3
Gemini 3.1 Pro59.866.172.9+13.1+6.8
Claude Sonnet 5.554.464.270.1+15.7+5.9

Under every judge, SolveEdit-Plan scores highest, ahead of both GVR and Direct, and the margins stay close to those of the main judge: Plan − Direct is 13.1 to 15.7 points (14.6 in the paper) and Plan − GVR is 5.9 to 6.8 points (6.3 in the paper). The margins do not shrink consistently when the judge differs from the planner: Gemini 3.1 Pro gives a larger margin over GVR than the main judge, and Claude Sonnet 5.5 a larger margin over Direct. Appendix F.8 reports the planning gain under all three judges.

The planner comparison in Q1 tests the same question from the other side: there the judge is fixed to GPT-5.6 Sol and the planner varies. If this judge favored instructions written by its own model, the GPT-5.6 Sol planner should score highest. It does not: the Claude Sonnet 5.5 planner scores higher (72.8 vs. 71.6), and the Gemini 3.1 Pro planner scores within 0.7 points (70.9).

Direct-evaluation rankings. We re-scored the Direct outputs of all 11 models with each judge.

ModelGPT-5.6 Sol (paper)Gemini 3.1 ProClaude Sonnet 5.5
GPT-Image-257.059.854.4
Seedream 5.0 Pro56.657.953.8
Gemini 3.1 Flash Image55.558.452.7
Qwen Image 2.0 Pro40.644.337.9
Kling V327.531.825.8
FLUX.2 [dev]22.827.321.4
Qwen-Image-Edit-250920.825.619.8
FLUX.1 Kontext [dev]18.723.417.6
BAGEL12.117.211.4
OmniGen29.714.79.1
HunyuanVideo-1.57.412.37.2
Kendall τ with main ranking–0.961.00

The judges differ in overall strictness: Gemini 3.1 Pro scores every model 1.3 to 5.1 points higher than the main judge, and Claude Sonnet 5.5 scores every model 0.2 to 2.8 points lower. GVR and SolveEdit-Plan shift in the same direction in the table above. The ranking is nevertheless stable. Claude Sonnet 5.5 reproduces it exactly. Gemini 3.1 Pro changes only one pair, swapping second and third place (Seedream 5.0 Pro and Gemini 3.1 Flash Image, which differ by 1.1 points under the main judge).

AuthorsOpenReview ↗

Part 2/3: Sensitivity to RA Noise (Q2) and RD Rule Types (Q3)

Q2. How sensitive are the regime gaps and rankings to RA scoring noise?

We agree that RA is relatively harder to judge than SA and VQ: relations such as connectivity, ordering and placement require tracing structure across the image. This is why a fixed subset of RA criteria is assigned to segmentation and detection tools before evaluation (Section 3.3).

The regime finding, however, does not rest mainly on RA. In Figure 5b, the largest domain-adjusted IS→RD drop is in required semantic accuracy (−25.0), almost twice the relational drop (−13.0). Abstentions are rare: evidence coverage is at least 98.8% for every model (Appendix D.1). They are excluded from the six diagnostic scores and add zero credit to R, so they cannot inflate completion.

To test RA noise directly, we rescored the full benchmark in two further ways: with all RA criteria removed (remaining weights renormalized), and with only the tool-verified RA criteria kept (geometry, topology, segmentation), which are the most reliable RA judgments.

ModelRD − IS (all criteria)RD − IS (RA removed)RD − IS (tool-verified RA only)
GPT-Image-2−23.7−23.6−20.5
Seedream 5.0 Pro−17.8−21.5−15.0
Gemini 3.1 Flash Image−17.2−21.8−14.8
Kendall τ with main ranking–1.001.00

Removing the dimension with the lowest human agreement leaves the gap unchanged or larger for all three models. Keeping only tool-verified RA criteria gives a gap of 14.8 to 20.5 points, still large for every model. The gap narrows in this setting because tool verification removes the failure cases on which the regimes differ most, leaving a harder but more homogeneous subset; on this subset the regimes move closer, while the overall ranking is preserved. The ranking of all 11 models is identical to the main ranking under both settings (Kendall τ = 1.00).

Q3. What rules are most common in RD, and can RD be solved without the image?

We tagged all 1,363 RD cases by rule type. The tags were assigned by a model from the case annotations.

Rule typeShare of RDGPT-Image-2 SolveScore
Reference relation (match an exemplar or adjacent structure)36.5%46.5
Explicit constraint23.7%48.4
Pattern or sequence11.4%51.3
Game or puzzle rule10.9%35.4
Legend10.5%58.2
Capacity or compatibility7.1%37.4

Game or puzzle rules and capacity constraints are the hardest for editors, and legends the easiest.

To test text-only solvability, a text-only model (Gemini 3.8 Flash) predicted the final state from the request alone for all 2,728 cases. It had to commit to concrete objects and positions, and a separate model graded each prediction against the contract.

RegimeText-only scoreFully correct
IS26.113.2%
SD25.112.8%
RD8.42.7%

RD is solvable from text in only 2.7% of cases, the lowest of the three regimes, and scores 17.7 points below IS. Even IS scores only 26.1, because the objects to edit are visible only in the image; RD additionally requires reading the rule from it. These are wrong guesses rather than refusals: at most 5.1% of predictions are left unspecified. The Text-only Rewrite control in the paper agrees with this: rewriting the request without the image adds only 0.3 points over Direct.

AuthorsOpenReview ↗

Part 3/3: Scale, Balance and Aspect Ratio (W2)

Scale. SolveEdit is the largest of the reasoning-editing test sets we compare against, and each case carries 10.8 atomic criteria on average (29,460 in total). More importantly, it measures properties that these benchmarks do not:

BenchmarkTest casesScoring unitPreservation scored byReference-free scoringHuman validationSource of deciding information annotated
SmartEdit (Reason-Edit), CVPR'24219pixel / CLIP metrics, human ratingbackground PSNR / SSIM / LPIPS✗ (reference for all cases)–✗
RISEBench, NeurIPS'253603 holistic 1–5 scoresone generic consistency scorepartial (reference for 70 / 360)score-level (MAE)✗
KRIS-Bench, NeurIPS'251,2674 holistic 1–5 scoresone generic consistency scorepartial (reference for 50 / 1,267)score-level (Pearson r)✗
UniREditBench, arXiv'252,7003 holistic 1–5 scores, weightedone generic consistency score✗ (reference for all cases)score-level (Pearson r)✗
VIBE, arXiv'261,034task-level binary sub-criteriaone binary context-preservation checkpartial (golden labels for some tasks)score-level (Pearson r)✗
SolveEdit2,728case-specific atomic criteriacase-specific protected criteria (14,629)✓ (all cases)criterion-level (3,832 judgments)✓ (IS / SD / RD)

"Reference-free" means correctness is judged without comparing the output to a reference image, so a valid output is not penalized for differing from one reference solution. Reference-image counts for SmartEdit, RISEBench and KRIS-Bench follow UniREditBench (Table 1).

Statistical power and balance. We agree that the roughly 50 cases per subdomain are too few for fine-grained claims, so all claims in the paper stay at the domain and regime level. RD is the largest regime because it covers the most varied rule types (Q3). To check that RD does not drive the leaderboard, we recomputed every model's score as a macro average with equal weight on IS, SD and RD. All 11 models keep exactly the same rank (Kendall τ = 1.00). Scores rise by 2.0 to 3.6 points because the hardest regime receives less weight.

ModelCase-weightedRegime-balanced
GPT-Image-257.060.2
Seedream 5.0 Pro56.658.6
Gemini 3.1 Flash Image55.557.6
Qwen Image 2.0 Pro40.642.6
Kling V327.529.8
FLUX.2 [dev]22.825.1
Qwen-Image-Edit-250920.823.4
FLUX.1 Kontext [dev]18.721.5
BAGEL12.115.3
OmniGen29.713.2
HunyuanVideo-1.57.411.0

Aspect ratio. The current release uses square inputs and 1024×1024 outputs for all image models. We now state this in the limitations (Appendix G); extending the benchmark to other aspect ratios is future work.

All new experiments in this response are included in Appendix F of the revised PDF (F.7 to F.11).

Reviewer_qsLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SolveEdit, a benchmark for evaluating the visual problem-solving capabilities of generative models. Unlike conventional image editing tasks that explicitly specify the required modifications, SolveEdit requires models to determine the necessary scene transformations from an input image and a goal description, while preserving unrelated content. The authors construct a benchmark containing 2,728 cases across 10 domains and 54 subdomains, and categorize the tasks into three types based on the information required to solve them: Instruction-Specified (IS), State-Dependent (SD), and Rule-Dependent (RD). To evaluate the generated results, the paper proposes SolveScore, an atomic-criterion-based metric that measures both task completion and the preservation of unrelated content. Experiments evaluate 9 image generation models and 2 video generation models, with the best-performing model, GPT-Image-2, achieving a SolveScore of 57.0%. The authors further introduce SolveEdit-Plan, a training-free approach that guides generation through explicit scene transformation planning, improving the score of GPT-Image-2 to 71.6% and achieving additional improvements on two other generators.

Critical strengths
  1. The paper categorizes visual editing tasks into IS, SD, and RD based on the source of information required to determine the transformation, and constructs corresponding evaluation cases across 10 application domains. For example, the circuit repair task in Figure 1 requires the model to determine where a connection should be made based on the circuit state, while Figure 2 includes molecular structure completion, path planning, and object arrangement under rule-based constraints. These tasks require models to understand the scene and determine appropriate modifications before generation, allowing the benchmark to evaluate capabilities that have rarely been examined separately in conventional instruction-driven image editing.

  2. SolveScore separately models task completion and the preservation of unrelated content, and further distinguishes semantic accuracy, relational accuracy, and visual quality. This enables more targeted evaluation when a model produces visually plausible results that nevertheless fail to satisfy task constraints. The authors also conduct human validation on 360 model-case pairs involving 3,832 atomic judgments, achieving 95.3% agreement between the automatic evaluator and human consensus. Together with the evaluation results across 11 generative models, this evaluation framework demonstrates practical utility and provides a foundation for further analysis of failure modes in generative models.

Critical weaknesses
  1. Section 3.1 divides the tasks into IS, SD, and RD, but the construction of these three categories does not establish controlled comparisons that strictly account for other factors. Section 5.3 and Figure 5 show that model performance decreases by 17.2–23.7 percentage points from IS to RD, which the authors interpret as evidence that models face greater difficulties when recovering scene states or interpreting visual rules. However, samples from different categories may also differ substantially in the number of targets, the complexity of spatial relationships, and the difficulty of editing operations. Although the authors adjust for domain distribution, controlling only for application domains is insufficient to rule out the influence of these factors. Furthermore, the paper does not report inter-annotator agreement for the IS, SD, and RD classifications, and the distinction between visual rules in RD and scene states in SD may be ambiguous for certain tasks.

  2. Section 5.3 interprets the substantial decreases in Required Semantic Accuracy and Required Relational Accuracy as evidence that models struggle to determine the correct scene transformations, but this conclusion lacks direct supporting evidence. Even when a model correctly understands the required operation, it may still fail these metrics because it cannot accurately generate object positions, quantities, or connections. In fact, Section 3.1 explicitly distinguishes between solution-recovery errors and visual-execution errors, yet the current SolveScore evaluates only the final outputs and cannot reliably distinguish these two types of errors. The observations in Figure 5 demonstrate that models fail to satisfy task constraints, but they do not directly establish that these failures primarily occur during the planning stage. Given that this diagnosis serves as the central motivation for SolveEdit-Plan, the lack of validation of intermediate transformation decisions weakens the connection between the experimental observations and the proposed method.

  3. Section 3.3 defines SolveScore as (\max(0,R-\lambda D)), with (\lambda=0.5). This means that, assuming the quality gate is passed, a generated result can still receive a score of 50% even if it satisfies all required conditions while violating every protected condition. However, Equation (2) in Section 3.1 requires a valid final result to satisfy both the target transformation and content preservation conditions, creating a discrepancy between the task formulation and the evaluation metric. Although the paper reports ranking stability under different values of (\lambda), it does not additionally report task success rates based on the simultaneous satisfaction of all critical conditions. Moreover, the 95.3% human agreement reported in Appendix C primarily indicates that the automatic evaluator can reproduce human judgments under predefined contracts, but it does not establish whether these contracts adequately cover the different valid solutions permitted by each task. Therefore, while the current experiments support the practical utility of SolveScore as a fine-grained evaluation metric, more direct validation is needed to determine whether it accurately reflects the final correctness of visual problem solving.

  4. In Section 5.4, SolveEdit-Plan improves the SolveScore of GPT-Image-2 from 57.0% to 71.6%, outperforming Generic Vision Rewrite by 6.3 percentage points. However, although the two methods use the same planning model and number of calls, their visual inputs are not fully matched. According to Appendix E, SolveEdit-Plan can additionally use up to six local image crops, whereas Generic Vision Rewrite only uses the original image. Therefore, the performance improvements may result from both more detailed local visual observations and explicit transformation planning. The ablation experiments in Figure 6 remove contrastive checking and preservation constraints, but do not separately examine the contribution of local image crops. More importantly, removing contrastive checking and preservation constraints reduces performance to 63.8% and 63.0%, respectively, both below the 65.3% achieved by Generic Vision Rewrite. These results suggest that the overall design is effective, but the independent contributions of its components remain unclear, as does the extent to which the improvements can be attributed to the scene transformation reasoning emphasized by the authors.

  5. The human validation in Appendix C primarily focuses on the agreement between the automatic evaluator and human judgments, where annotators are also required to make decisions based on the atomic criteria predefined by the authors. Although the paper reports 95.3% agreement at the atomic level and 93.6% reasonable or partially reasonable judgments at the case level, these results do not sufficiently demonstrate that the atomic criteria themselves accurately cover all valid solutions to a task. The paper highlights support for multiple valid final states as an important feature of SolveScore, but lacks dedicated experiments evaluating different valid editing outcomes for the same task. It also does not provide an independent assessment of task completion that does not rely on the predefined contracts. If these contracts omit certain valid solutions or fail to fully capture task requirements, the resulting scores may still deviate from the models' actual problem-solving capabilities. This concern affects my assessment of the validity of the benchmark evaluation.

  6. Appendix E shows that the Resolve stage of SolveEdit-Plan can additionally receive up to six local image crops selected during the Inspect stage, whereas Generic Vision Rewrite only uses the original image. Although both methods use the same planning model and number of calls, the visual information available to them is not fully equivalent. Therefore, the 6.3-percentage-point improvement of SolveEdit-Plan over Generic Vision Rewrite in Figure 6 cannot be entirely attributed to explicit scene transformation planning, as part of the gain may come from the additional visual information provided by local crops. The existing ablation experiments mainly remove contrastive checking and content preservation constraints, without separately analyzing the contribution of local visual observations. Furthermore, the paper does not directly evaluate whether the intermediate transformations determined by the planner are correct, leaving the respective roles of planning and final generation insufficiently understood. A controlled experiment with strictly matched visual inputs would provide stronger evidence for the authors' explanation of the performance improvements.

Questions for authors

Please refer to Critical Weaknesses.

9 comments
AuthorsOpenReview ↗

Part 1/4: Confounds Across Regimes and Label Reliability (W1)

We thank the reviewer for the detailed and constructive review. We answer the weaknesses in order. W4 and W6 both concern crops and component contributions, so we answer them together. All new experiments below are included in Appendix F of the revised PDF, and changed text is shown in blue. Unless stated otherwise, every automatic experiment below runs on the full 2,728-case benchmark with GPT-Image-2 as the fixed editor (one generation per case) and the same judge and evaluation prompt as Section 5.4.

W1. Confounds across regimes, and label reliability.

We agree that domain adjustment alone cannot rule out differences in target count, relational complexity or operation difficulty. To hold these factors fixed, we ran an oracle-transition experiment. For each case, we rewrite the required criteria of the hidden contract as an explicit instruction that names the targets, the action, the destination and the final state. Protected criteria are not included. The same editor then generates once per case. The oracle uses the answer key, so it is not a competing method. It measures how the editor performs once determining the transition is no longer its job.

ConditionISSDRDAllRD − IS
Direct70.663.146.957.0−23.7
SolveEdit-Plan79.177.864.771.6−14.4
Oracle transition84.586.082.583.8−2.0

The comparison is within the same cases. Between Direct and Oracle, the image, the number of targets, the spatial relations and the editing operation are identical; the only change is whether the editor must determine the transition zz. If RD cases trailed IS because they had more targets, more complex relations or harder operations, those factors would still be present under Oracle, and RD would still trail IS. Instead, the gap falls from 23.7 to 2.0 points.

One caveat: the oracle instruction is written from the same required criteria that are scored, so its absolute score is optimistic, and IS also gains 13.9 points. This alignment applies equally to every regime. We therefore draw the conclusion from the difference between regimes: the oracle closes 21.7 of the 23.7-point RD − IS gap.

Label reliability. Three independent annotators relabeled 300 regime-stratified cases without seeing the released labels or contracts. Agreement is almost perfect (Fleiss' κ = 0.88), and the majority vote matches the released label on 273/300 cases (91.0%). Most of the remaining disagreements (17 of 27) lie on the SD/RD border the reviewer mentions, where the deciding scene state also follows a visible rule. The main gap, between IS and RD, involves two regimes that are confused only once in 300 cases, so label noise cannot account for it. Full statistics and the confusion matrix are in our response to Reviewer BnGd (Q2).

AuthorsOpenReview ↗

Part 2/4: Solution Recovery vs. Visual Execution (W2) and Strict Success (W3)

W2. Do failures come from solution recovery or from visual execution?

We agree that the inference in Section 5.3 was indirect. Final-output scores alone cannot separate the two error types, as our limitations section notes, but the main text states the conclusion more strongly than the evidence supports. We add two analyses.

(1) Oracle decomposition (table in W1). Supplying the correct transition raises SolveScore from 57.0 to 83.8. The 16.2 points still missing under Oracle are errors the editor makes when it already knows the transition, i.e., visual-execution and preservation errors. These errors are substantial, which confirms the reviewer's point. They are, however, nearly flat across regimes (84.5 / 86.0 / 82.5), whereas the Direct scores fall sharply from IS to RD. The regime gap in Figure 5 is therefore mainly a solution-recovery effect, while execution errors account for a roughly constant share in every regime. SolveEdit-Plan recovers 14.6 of the 26.8 points that the oracle recovers, and the remaining distance to the oracle is largest on RD (17.8 points, vs. 5.4 on IS and 8.2 on SD).

(2) Direct audit of intermediate transitions. Annotators judged whether the transition z^\hat z selected by SolveEdit-Plan (its targets, action and final state) is admissible for the case, on 300 regime-stratified cases.

RegimenAdmissiblePartially admissibleWrong
IS10086104
SD10073216
RD10069229
All300228 (76.0%)53 (17.7%)19 (6.3%)

Planner accuracy falls from IS to RD in the same order as the editors' scores, which directly links the observed regime gap to transition determination.

We have revised the wording in Section 5.3 accordingly; both analyses are in Appendix F.2 and F.3.

W3. SolveScore vs. Eq. (2).

The reviewer is right that the two differ. Eq. (2) defines validity as a binary property, while SolveScore gives graded credit so that near-misses can be told apart from wrong transitions. Strict success is the binary version of Eq. (2) under the contract: the output passes the quality gate and every required and protected criterion passes (partial counts as failure).

GPT-Image-2 conditionSolveScoreStrict success
Direct57.012.0%
Text-only Rewrite57.312.3%
GVR65.314.2%
SolveEdit-Plan71.617.2%
Oracle transition83.822.0%

Both metrics order the conditions identically. Strict success is low in absolute terms because a case has 10.8 atomic criteria on average and all of them must pass. Even with the oracle transition, only 22.0% of outputs are perfect. This is why we use the graded score as the primary metric: the binary metric compresses most differences between methods into a few points. Strict success is now reported alongside SolveScore in Appendix F.4.

On λ: the value 0.5 was fixed before evaluation and encodes completion as the primary objective. The sweep in Appendix B.6 over λ ∈ [0, 1] keeps the leaderboard rank correlation at 0.909 or above. At λ = 1 the score is R − D, so complete damage cancels complete completion, which removes the case the reviewer describes. The validity of the contracts themselves is addressed under W5.

AuthorsOpenReview ↗

Part 3/4: Crops and Component Contributions (W4/W6)

W4 / W6. Crops and component contributions.

As noted in our response to Reviewer BnGd, the primary contribution of the paper is the benchmark and its evaluation protocol; SolveEdit-Plan is a deliberately simple probe showing that the diagnosed gap is actionable. The analyses below examine which of its steps matter.

(1) Matched visual input. We gave GVR exactly the crops that SolveEdit-Plan's Inspect stage requested for each case. GVR + crops scores 63.5, which is 1.8 points below GVR without crops. With identical visual input, SolveEdit-Plan leads by 8.1 points (71.6 vs. 63.5). Crops help SolveEdit-Plan because each one is requested to answer a specific open variable (e.g., which slot is free), and the answer is used to choose between candidate edits. GVR receives the same pixels without the question behind them and makes no explicit choice that they could inform, so the close-ups mostly add local detail to its instruction and point the editor at regions the goal does not require it to change (see our response to Reviewer BnGd, Q1).

(2) Component factorial. We also added each component individually to GVR, using the same two calls. Together with the paper's ablations, this covers all eight combinations of crops, contrastive checking and preservation inference:

CropsContrastPreservationSolveScoreCorresponds to
–––65.3GVR
–✓–67.5GVR + contrast
––✓66.8GVR + preservation
–✓✓66.1SolveEdit-Plan w/o crops
✓––63.5GVR + crops
✓–✓63.8SolveEdit-Plan w/o contrast
✓✓–63.0SolveEdit-Plan w/o preservation
✓✓✓71.6SolveEdit-Plan

The components do not have separable, additive effects. Without crops, the structured steps add 0.8 to 2.2 points over GVR. With crops, every configuration that lacks a step scores below GVR (63.0 to 63.8), and only the full procedure reaches 71.6. This explains the reviewer's observation that both single ablations fall below GVR. The completion and damage scores in Appendix E show how each step fails without the others:

  • Without contrastive checking, R falls from 75.8 to 69.1, while D stays low (11.5, below GVR's 16.3). The instruction is well scoped, but Resolve commits to the first plausible transition without comparing it against alternatives, so a specific but wrong edit is carried out faithfully.
  • Without preservation inference, D rises from 8.6 to 18.1, above GVR's 16.3, and R also falls to 68.0. The instruction specifies the change but not its scope, so the editor alters surrounding content, including content that the required conditions depend on.

In short, the gain requires evidence gathering, candidate comparison and scoping together. The factorial is in Appendix F.5, and Section 5.4 now reports the matched-crop comparison.

(3) Intermediate transitions are now audited directly (W2): 76.0% of the transitions selected by the planner are admissible and 17.7% are partially admissible.

Reviewer_BnGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a 2,728-case visual editing benchmark, an evaluation score for task completion and unintended changes, and a two-stage VLM workflow that writes prompts for existing image generators.

Critical strengths

Separating whether an edit achieves its goal from whether it damages unrelated content is useful. The annotated cases provide a resource for studying both.

Critical weaknesses
  1. The planner is essentially a two-call GPT-5.6 Sol workflow: inspect the image, request crops, and write a better editing prompt. I see limited methodological novelty here. The improvement over generic visual rewriting is 65.3 → 71.6, but the proposed method gets extra crops and the baseline does not. This comparison does not establish how much the proposed analysis steps actually help.

  2. The benchmark’s main new distinction is IS/SD/RD, but the paper does not report how consistently independent annotators assign these labels. The reported 95.3% human agreement concerns scoring model outputs, not validating these categories. This leaves an important part of the benchmark insufficiently supported.

Questions for authors
  1. Does the improvement over generic rewriting remain when both methods receive the same crops?
  2. What is the independent annotator agreement on IS/SD/RD labels?
5 comments
AuthorsOpenReview ↗

Part 1/3: Scope of the Contribution and What SolveEdit Measures

We thank the reviewer for the two concrete questions. We first clarify the scope of the paper's contribution, then answer both questions with new experiments. Unless stated otherwise, every automatic experiment below runs on the full 2,728-case benchmark with GPT-Image-2 as the fixed editor (one generation per case) and the same judge and evaluation prompt as Section 5.4.

Scope of the contribution. We would like to clarify how SolveEdit-Plan fits into the paper. The primary contribution is the benchmark and its evaluation protocol. They measure an ability that existing editing benchmarks do not isolate: working out which edit a goal requires from the image itself, then making that edit without disturbing the rest of the scene. SolveEdit-Plan is kept deliberately simple. Its role is to show that the gap the benchmark reveals is real and can be partly closed. Part 2 explains why its design follows from the task formulation, and why its gain does not come from prompt rewriting alone.

In standard editing benchmarks, the instruction states the change ("make the car red"), so they measure whether a model can carry out a given edit. Many practical requests state only a goal: "repair the circuit so the lamp lights", "move the piece to a free square", "complete the molecule". The correct edit then depends on what the image shows: which terminal is open, which square is free, which rule the legend defines. Recent reasoning-editing benchmarks add reasoning to the instruction. They do not, however, record where the information that fixes the edit comes from, and they score each output against a reference image or with generic criteria shared across a task. SolveEdit is built for the goal-only setting:

  • every request hides the fact that decides the edit;
  • the IS/SD/RD axis records whether that fact comes from the request, the current scene state or a rule shown in the image;
  • atomic contracts state the conditions that any valid final state must satisfy, so they accept multiple valid solutions, and they score collateral damage separately from completion.

The results show that current editors handle this setting poorly. The strongest editor reaches 57.0, falling from 70.6 on IS cases to 46.9 on RD cases. An oracle upper bound locates the bottleneck: when we write the correct edit into the instruction, the same editor scores 84.5 on IS and 82.5 on RD. The oracle is not a method, because it uses the answer key. Its absolute level is also optimistic, because the instruction is written from the scored criteria, but this applies equally to every regime. Current editors can largely render these edits once told what they are. What they lack is the ability to determine the edit from the image, and a benchmark whose instructions already specify the edit cannot detect this gap.

AuthorsOpenReview ↗

Part 2/3: Role of SolveEdit-Plan and Matched-Crop Comparison (Q1)

Role and design of SolveEdit-Plan. SolveEdit-Plan is intentionally lightweight. It needs no training and leaves the editor unchanged, so any gain must come from determining the edit, not from added model capacity. Its steps are not a generic recipe for better prompts. Each one implements an element of the formulation in Section 3.1:

  • Inspect identifies which parts of the transition zz the request leaves open (target, action, destination, final state, preservation scope) and collects visual evidence for each.
  • Resolve compares candidate transitions against that evidence and compiles the chosen one into completion and preservation obligations.

The reviewer's description of "writing a better editing prompt" corresponds to our Generic Vision Rewrite (GVR) control: same model, same two calls, same image, and the explicit goal of producing a better instruction. SolveEdit-Plan exceeds GVR by 6.3 points, and by 8.1 when both receive the same crops (Q1 below). The gain is distributed as the formulation predicts. Over Direct, it is +8.5 on IS, +14.7 on SD and +17.8 on RD, where the open variable is hardest to recover. It also holds beyond one configuration:

  • without modification on other editors: +6.1 on Qwen-Image-Edit-2509 and +6.6 on HunyuanVideo-1.5;
  • over a same-backbone GVR control for each of four planner backbones (GPT-5.6 Sol, Gemini 3.1 Pro, Claude Sonnet 5.5 and the open-weight Qwen3.5-27B), by 3.3 to 6.3 points.

We regard this simplicity as useful for a benchmark paper. A simple, transparent procedure that consistently beats a strong same-budget control shows that the benchmark's diagnosis is actionable, and gives future methods a clear reference point. We have made this role explicit in Section 4 of the revised PDF.

Q1. Does the improvement remain when both methods receive the same crops?

Yes, and the margin grows. In GVR + same crops, GVR receives exactly the crops that SolveEdit-Plan requested for each case. In SolveEdit-Plan w/o crops, the planner sees only the source image. Model, call count and generation budget are unchanged.

MethodCropsSolveScore
Directnone57.0
Generic Vision Rewrite (GVR)none65.3
GVR + same crops as SolveEdit-Plan✓63.5
SolveEdit-Plan w/o cropsnone66.1
SolveEdit-Plan✓71.6

With identical visual input, SolveEdit-Plan leads GVR by 8.1 points (71.6 vs. 63.5), up from 6.3 in the paper. This comparison is now in Section 5.4 and Appendix F.5 of the revised PDF.

Why the crops help SolveEdit-Plan. SD and RD cases are often decided by small details: an empty slot, an unconnected wire end, a symbol in a legend, a mark on a board. In the full image these details cover few pixels and are easy to misread. SolveEdit-Plan first names the variable it cannot yet resolve (for example, which slot is free), then requests a crop of the region that answers it, and records the answer. The second call compares candidate edits against these recorded answers. Targeted inspection is therefore part of the procedure, not an extra input: the same crops lower GVR's score (63.5), whereas SolveEdit-Plan uses them to answer the variable it has named and to choose between candidate edits. What helps is deciding which detail resolves the edit, and then looking at it.

Why the same crops do not help GVR. GVR receives the same pixels but not the question each crop was taken to answer. Without that question, a close-up of one region carries no information about which detail decides the edit, and GVR makes no explicit choice between candidate edits that the crop could inform. GVR's task is to write a better instruction, so extra close-ups tend to enter the instruction as additional local detail. This points the editor at regions the goal may not require it to change. The component factorial in our response to Reviewer qsLi (W4) is consistent with this: crops lower the score in every configuration except the full procedure, where the planner both uses them to choose the edit and limits the edit's scope.

AuthorsOpenReview ↗

Part 3/3: Independent Agreement on IS/SD/RD Labels (Q2)

Q2. What is the independent annotator agreement on IS/SD/RD labels?

The regime labels are highly reproducible: three independent annotators reach Fleiss' κ = 0.88, which is in the "almost perfect" range (κ > 0.80; Landis & Koch, 1977), and their majority vote matches the released label on 91.0% of cases.

The 95.3% figure in the paper concerns scoring, so we ran a separate labeling study for the regimes. Three annotators who did not take part in benchmark construction relabeled 300 cases: 100 per regime, sampled in proportion to domain within each regime. They saw the image, the request, one admissible final state and the decision protocol of Appendix A.4. They did not see the released labels, the contracts or the rule sources, and case order was randomized per annotator.

MetricValue
Fleiss' κ (3 annotators)0.88
Pairwise Cohen's κ0.86 / 0.88 / 0.89
Each annotator vs. released label (Cohen's κ)0.87 / 0.89 / 0.88
All three annotators agree265/300 (88.3%)
A majority (≥ 2 of 3) exists298/300 (99.3%)
Majority vote matches released label273/300 (91.0%), κ = 0.87
Released label ↓ / Majority vote →ISSDRDNo majority
IS (100)95401
SD (100)38881
RD (100)19900

The remaining disagreements sit between adjacent regimes. Of the 27 mismatches, 17 lie on the SD/RD border, typically where the deciding scene state also follows a visible rule. IS and RD, the two regimes that differ most, are confused only once in 300 cases. This study and its protocol are now in Appendix F.1 of the revised PDF.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Discussion and Revisions

We thank the AC and all reviewers for their time and constructive feedback. We were glad that the reviewers found the problem formulation clean and the IS/SD/RD axis "not systematically isolated in prior editing benchmarks" (dGtr), the separation of task completion from collateral damage useful (BnGd, qsLi), and the contract-based scoring free of reference-image bias (dGtr).

We have uploaded a revised PDF. Changed text is shown in blue. All new experiments are collected in a new Appendix F; Appendices A–E keep their original numbering, so the references in our earlier responses remain valid.

Main concerns and how we addressed them. Unless stated otherwise, all new experiments run on the full 2,728-case benchmark.

  1. Reliability of IS/SD/RD labels (BnGd, qsLi). Three independent annotators relabeled 300 cases without seeing the released labels: Fleiss' κ = 0.88, and the majority vote matches the released label on 91.0% of cases. IS and RD are confused only once in 300 cases. [F.1]
  2. Whether regime gaps reflect transition determination or other confounds (qsLi). An oracle that supplies the correct transition on the same cases shrinks the RD−IS gap from 23.7 to 2.0 points. A human audit of planner-selected transitions shows accuracy falling from IS to RD in the same order as the editors' scores. We have revised Section 5.3 to state this conclusion more carefully. [F.2, F.3]
  3. Source of SolveEdit-Plan's gain (BnGd, qsLi). With identical crops, SolveEdit-Plan leads the generic rewrite control by 8.1 points (71.6 vs. 63.5), and the same crops lower the control's score. A 2³ component factorial shows that the gain requires evidence gathering, candidate comparison and scoping together. [Section 5.4, F.5]
  4. Planner generality and planner–judge alignment (dGtr). SolveEdit-Plan beats a same-backbone control for four planners, including an open-weight 27B model (+3.3 to +6.3). Under judges from two other vendors, the planning gain stays within 13.1–15.7 points and the 11-model ranking is preserved (Kendall τ ≥ 0.96). [F.7, F.8]
  5. Validity of the contracts and of SolveScore (qsLi, dGtr). Alternative valid solutions score 87.6 vs. 21.8 for invalid ones; contract-free human judgments of goal achievement reproduce the method ordering; strict success orders all conditions identically to SolveScore; and the model ranking is unchanged when relational criteria are removed or regimes are equally weighted. [F.4, F.6, F.9, F.11]

Scope. The primary contribution of the paper is the benchmark and its evaluation protocol. SolveEdit-Plan is intentionally simple and training-free. It shows that the gap the benchmark identifies can be partly closed, and it gives future methods a reference point. Section 4 now states this design choice.

Other revisions. We rephrased the planner failure-rate claim (Appendix E) and added two limitations: contracts are not exhaustive, and the current release uses square inputs (Appendix G). We will simplify the notation of Section 3 in the final version.

We thank the reviewers again for helping us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.