- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SolveEdit, a benchmark for visual problem solving through scene transformation. As authors claimed, existing image-editing benchmarks evaluate whether a model can execute an explicitly specified change. The proposed SolveEdit instead asks the model to infer what change is needed from the source image and a high-level goal. The benchmark contains 2,728 cases across 10 domains and 54 subdomains, organized by the information source that determines the required transition: Instruction-Specified (IS, 736 cases), State-Dependent (SD, 629 cases), and Rule-Dependent (RD, 1,363 cases). Each case has an atomic transition contract with required conditions and protected conditions. SolveScore is used to measure completion minus a bounded penalty for collateral damage. Authors evaluate 11 models (9 image-to-image, 2 image-to-video) and find that the strongest model achieves only 57.0% SolveScore. They further introduce SolveEdit-Plan, a two-stage test-time planner that instantiates the transition before a single call to an unchanged editor. Under matched budget, SolveEdit-Plan improves GPT-Image-2 from 57.0% to 71.6%, outperforming a call-matched generic rewrite control by 6.3 points, with larger gains on SD (+23.3%) and RD (+38.0%) than on IS (+12.0%).
Critical strengths
- This paper has a clean problem formulation and orthogonal evaluation axis. The IS/SD/RD decomposition separates instruction following from scene understanding and rule inference within a single generative task. This is orthogonal to content domain and task type. The same spatial operation can land in any regime depending on what fixes the valid outcome. This axis has not been systematically isolated in prior editing benchmarks.
- The contract-based scoring avoids reference-image bias. By specifying required and protected atomic conditions rather than matching a single reference output, SolveScore accommodates multiple valid solutions and separately measures completion and collateral damage. The six diagnostic cells provide interpretable failure attribution.
Critical weaknesses
- The planner generalization is untested. The planning experiments use only GPT-5.6 Sol as the planner, and the 14.6% gain may largely reflect this model's strong visual reasoning ability rather than the planning procedure itself. No other planner backbone is tested. It is unclear whether smaller or lightweight planner models can achieve similar gains, or whether the procedure transfers at all. The 0% planner failure rate reported by the authors likely reflects GPT-5.6 Sol's capability rather than task ease. Additionally, since both the planner and the evaluator use GPT-5.6 Sol, the absolute SolveScore improvement may carry some systematic alignment between plan and judge, though this does not affect the model ranking under direct evaluation.
- The benchmark size and distribution looks somewhat imbalanced. With 2,728 cases, the benchmark is modest in scale. RD accounts for 1,363 cases while IS has only 736. The per-subdomain average is about 50 cases, which limits statistical power for fine-grained claims. While all evaluated outputs use 1024×1024 resolution, the coverage of different aspect ratios is not discussed.
Questions for authors
- SolveEdit-Plan uses only GPT-5.6 Sol as the planner. Have authors ever tested the planning procedure with a smaller or lightweight planner model on a subset? Do the gains persist, or do they largely reflect GPT-5.6 Sol's raw visual reasoning ability?
- Table 8 shows RA human agreement is only 91.8% (vs. 96.6% SA, 97.3% VQ). Since the paper's main diagnostic claim is that relational conditions are the dominant failure mode, how sensitive are the regime gaps (Figure 5b) and the model rankings to RA scoring noise? Have authors re-estimated the main findings after excluding criteria where the evaluator abstains or where human agreement is lowest?
- The RD accounts for 50% of the benchmark. What types of in-image rules are most common (legends, constraints, repeated patterns, game rules)? Are the RD cases solvable by a strong text-only LLM without the image, or do they genuinely require visual rule reading?
- Both SolveEdit-Plan and SolveScore use GPT-5.6 Sol. Does this create a systematic alignment between what the planner produces and what the evaluator rewards? Have authors compared direct model rankings or planning gains when evaluated with a different VLM judge?
5 comments
Part 2/3: Sensitivity to RA Noise (Q2) and RD Rule Types (Q3)
Q2. How sensitive are the regime gaps and rankings to RA scoring noise?
We agree that RA is relatively harder to judge than SA and VQ: relations such as connectivity, ordering and placement require tracing structure across the image. This is why a fixed subset of RA criteria is assigned to segmentation and detection tools before evaluation (Section 3.3).
The regime finding, however, does not rest mainly on RA. In Figure 5b, the largest domain-adjusted IS→RD drop is in required semantic accuracy (−25.0), almost twice the relational drop (−13.0). Abstentions are rare: evidence coverage is at least 98.8% for every model (Appendix D.1). They are excluded from the six diagnostic scores and add zero credit to R, so they cannot inflate completion.
To test RA noise directly, we rescored the full benchmark in two further ways: with all RA criteria removed (remaining weights renormalized), and with only the tool-verified RA criteria kept (geometry, topology, segmentation), which are the most reliable RA judgments.
| Model | RD − IS (all criteria) | RD − IS (RA removed) | RD − IS (tool-verified RA only) |
|---|---|---|---|
| GPT-Image-2 | −23.7 | −23.6 | −20.5 |
| Seedream 5.0 Pro | −17.8 | −21.5 | −15.0 |
| Gemini 3.1 Flash Image | −17.2 | −21.8 | −14.8 |
| Kendall τ with main ranking | – | 1.00 | 1.00 |
Removing the dimension with the lowest human agreement leaves the gap unchanged or larger for all three models. Keeping only tool-verified RA criteria gives a gap of 14.8 to 20.5 points, still large for every model. The gap narrows in this setting because tool verification removes the failure cases on which the regimes differ most, leaving a harder but more homogeneous subset; on this subset the regimes move closer, while the overall ranking is preserved. The ranking of all 11 models is identical to the main ranking under both settings (Kendall τ = 1.00).
Q3. What rules are most common in RD, and can RD be solved without the image?
We tagged all 1,363 RD cases by rule type. The tags were assigned by a model from the case annotations.
| Rule type | Share of RD | GPT-Image-2 SolveScore |
|---|---|---|
| Reference relation (match an exemplar or adjacent structure) | 36.5% | 46.5 |
| Explicit constraint | 23.7% | 48.4 |
| Pattern or sequence | 11.4% | 51.3 |
| Game or puzzle rule | 10.9% | 35.4 |
| Legend | 10.5% | 58.2 |
| Capacity or compatibility | 7.1% | 37.4 |
Game or puzzle rules and capacity constraints are the hardest for editors, and legends the easiest.
To test text-only solvability, a text-only model (Gemini 3.8 Flash) predicted the final state from the request alone for all 2,728 cases. It had to commit to concrete objects and positions, and a separate model graded each prediction against the contract.
| Regime | Text-only score | Fully correct |
|---|---|---|
| IS | 26.1 | 13.2% |
| SD | 25.1 | 12.8% |
| RD | 8.4 | 2.7% |
RD is solvable from text in only 2.7% of cases, the lowest of the three regimes, and scores 17.7 points below IS. Even IS scores only 26.1, because the objects to edit are visible only in the image; RD additionally requires reading the rule from it. These are wrong guesses rather than refusals: at most 5.1% of predictions are left unspecified. The Text-only Rewrite control in the paper agrees with this: rewriting the request without the image adds only 0.3 points over Direct.
Part 3/3: Scale, Balance and Aspect Ratio (W2)
Scale. SolveEdit is the largest of the reasoning-editing test sets we compare against, and each case carries 10.8 atomic criteria on average (29,460 in total). More importantly, it measures properties that these benchmarks do not:
| Benchmark | Test cases | Scoring unit | Preservation scored by | Reference-free scoring | Human validation | Source of deciding information annotated |
|---|---|---|---|---|---|---|
| SmartEdit (Reason-Edit), CVPR'24 | 219 | pixel / CLIP metrics, human rating | background PSNR / SSIM / LPIPS | ✗ (reference for all cases) | – | ✗ |
| RISEBench, NeurIPS'25 | 360 | 3 holistic 1–5 scores | one generic consistency score | partial (reference for 70 / 360) | score-level (MAE) | ✗ |
| KRIS-Bench, NeurIPS'25 | 1,267 | 4 holistic 1–5 scores | one generic consistency score | partial (reference for 50 / 1,267) | score-level (Pearson r) | ✗ |
| UniREditBench, arXiv'25 | 2,700 | 3 holistic 1–5 scores, weighted | one generic consistency score | ✗ (reference for all cases) | score-level (Pearson r) | ✗ |
| VIBE, arXiv'26 | 1,034 | task-level binary sub-criteria | one binary context-preservation check | partial (golden labels for some tasks) | score-level (Pearson r) | ✗ |
| SolveEdit | 2,728 | case-specific atomic criteria | case-specific protected criteria (14,629) | ✓ (all cases) | criterion-level (3,832 judgments) | ✓ (IS / SD / RD) |
"Reference-free" means correctness is judged without comparing the output to a reference image, so a valid output is not penalized for differing from one reference solution. Reference-image counts for SmartEdit, RISEBench and KRIS-Bench follow UniREditBench (Table 1).
Statistical power and balance. We agree that the roughly 50 cases per subdomain are too few for fine-grained claims, so all claims in the paper stay at the domain and regime level. RD is the largest regime because it covers the most varied rule types (Q3). To check that RD does not drive the leaderboard, we recomputed every model's score as a macro average with equal weight on IS, SD and RD. All 11 models keep exactly the same rank (Kendall τ = 1.00). Scores rise by 2.0 to 3.6 points because the hardest regime receives less weight.
| Model | Case-weighted | Regime-balanced |
|---|---|---|
| GPT-Image-2 | 57.0 | 60.2 |
| Seedream 5.0 Pro | 56.6 | 58.6 |
| Gemini 3.1 Flash Image | 55.5 | 57.6 |
| Qwen Image 2.0 Pro | 40.6 | 42.6 |
| Kling V3 | 27.5 | 29.8 |
| FLUX.2 [dev] | 22.8 | 25.1 |
| Qwen-Image-Edit-2509 | 20.8 | 23.4 |
| FLUX.1 Kontext [dev] | 18.7 | 21.5 |
| BAGEL | 12.1 | 15.3 |
| OmniGen2 | 9.7 | 13.2 |
| HunyuanVideo-1.5 | 7.4 | 11.0 |
Aspect ratio. The current release uses square inputs and 1024×1024 outputs for all image models. We now state this in the limitations (Appendix G); extending the benchmark to other aspect ratios is future work.
All new experiments in this response are included in Appendix F of the revised PDF (F.7 to F.11).
Part 1/3: Planner Backbones (Q1) and Planner–Judge Alignment (Q4)
We thank the reviewer for the positive assessment and the precise questions. Unless stated otherwise, every experiment below runs on the full 2,728-case benchmark with GPT-Image-2 as the fixed editor (one generation per case) and the same evaluation prompt as Section 5.4.
Q1. Do the gains persist with other planners?
Yes. We reran the planning comparison with three additional planners: two closed models from other vendors and one open-weight model. To separate the procedure from the backbone's raw ability, each planner is also run as a Generic Vision Rewrite (GVR) on the same backbone.
Direct is 57.0 for all rows. SolveEdit-Plan beats the same-backbone GVR for every planner, so the procedure adds value beyond what each backbone achieves by rewriting the prompt. The reviewer is also right that planner capability matters: with Qwen3.5-27B the gain over Direct is about half that of the closed planners. This is consistent with the benchmark's diagnosis, since the procedure helps to the extent that the planner can determine the transition from the image. We have not tested planners smaller than 27B. We agree that the 0% planner failure rate reflects GPT-5.6 Sol's capability rather than task ease, and have rephrased it in Appendix E. These results are in Appendix F.7.
Q4. Does using GPT-5.6 Sol as both planner and judge create alignment?
The design limits the channel for alignment. The planner never sees the contract, and the judge never sees the planner's text: it scores the generated image against the source image, the request and the contract. To test for residual alignment, we re-scored the same images with judges from two other vendors.
Planning gains.
Under every judge, SolveEdit-Plan scores highest, ahead of both GVR and Direct, and the margins stay close to those of the main judge: Plan − Direct is 13.1 to 15.7 points (14.6 in the paper) and Plan − GVR is 5.9 to 6.8 points (6.3 in the paper). The margins do not shrink consistently when the judge differs from the planner: Gemini 3.1 Pro gives a larger margin over GVR than the main judge, and Claude Sonnet 5.5 a larger margin over Direct. Appendix F.8 reports the planning gain under all three judges.
The planner comparison in Q1 tests the same question from the other side: there the judge is fixed to GPT-5.6 Sol and the planner varies. If this judge favored instructions written by its own model, the GPT-5.6 Sol planner should score highest. It does not: the Claude Sonnet 5.5 planner scores higher (72.8 vs. 71.6), and the Gemini 3.1 Pro planner scores within 0.7 points (70.9).
Direct-evaluation rankings. We re-scored the Direct outputs of all 11 models with each judge.
The judges differ in overall strictness: Gemini 3.1 Pro scores every model 1.3 to 5.1 points higher than the main judge, and Claude Sonnet 5.5 scores every model 0.2 to 2.8 points lower. GVR and SolveEdit-Plan shift in the same direction in the table above. The ranking is nevertheless stable. Claude Sonnet 5.5 reproduces it exactly. Gemini 3.1 Pro changes only one pair, swapping second and third place (Seedream 5.0 Pro and Gemini 3.1 Flash Image, which differ by 1.1 points under the main judge).