- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Contract-Guided Test-Time Scaling (CG-TTS), a training-free pipeline for instruction-guided image editing.
- Contract construction. GPT-4o, LISA, and Grounding DINO build an explicit "edit contract" containing the target, requested change, allowed and protected regions, and mandatory checks.
- Candidate generation. A frozen inpainting editor (BrushNet or SDXL-Inpainting) generates four candidates, with Source-Memory Routing (SMR) reusing cached source features.
- Verification. A staged verifier accepts a candidate only if all mandatory spatial and semantic checks pass.
- Repair. If no candidate passes, failure diagnosis (e.g., under-edit or over-edit) guides one repair round of four more candidates, so each input uses at most 8 editor calls.
On ReasonEdit and HumanEdit-core, CG-TTS improves background PSNR (e.g., 25.4→38.0 dB on ReasonEdit-U with BrushNet) and MLLM-judged instruction compliance over the base editors and over scalar Best-of-8 selection from the same candidate pool.
Critical strengths
- Explicit verification instead of a single scalar score. Replacing a scalar ranking score with requirement-level PASS/FAIL/UNCERTAIN checks is a sensible way to avoid candidates that look good overall but omit a requested change or alter protected content. Under the same candidate pool, it outperforms CLIP, VIEScore, and EditScore Best-of-8 selection (Tables 1-2).
- Budget-aware repair. Repair is triggered only on failure (32/131 ReasonEdit-U examples), giving 4.98 average editor calls, 37.8% fewer than uniform Best-of-8.
Critical weaknesses
W1. The method is a composition of existing components, and the framing as "test-time scaling" is not well justified.
- Unclear "scaling". For editing, more computation does not by itself yield more reliable results, just as more NFE does not. Here, the gains come mainly from external tools: GPT-4o contracts, LISA and Grounding DINO masks, and an MLLM-assisted verifier. The extra editor calls themselves contribute comparatively little. The paper does not show a scaling behavior, i.e., performance that keeps improving with the budget, that would justify the TTS framing.
- Limited novelty of components. Reusing cached source features of a frozen editor, as in SMR, is common in training-free editing (e.g., attention/feature injection in Prompt-to-Prompt, MasaCtrl, and inversion-based editors). It is not clear that this warrants "we introduce."
- Overall assessment. The work reads as a compositional inference pipeline rather than a new method.
- Fragile spatial contract. The spatial contract depends entirely on LISA and Grounding DINO, both 2024 models with known localization errors and biases. The paper does not analyze what happens when they mislocalize or miss the target, nor how often this occurs. Such errors propagate to every downstream stage, since the same contract is used for generation, verification, and repair.
- Failure diagnosis assumes a single failure type. Each candidate is assigned one diagnosed failure type (Eq. 7). Real failures often co-occur, e.g., over-edit together with semantic mismatch, and some instructions are hard for the editor to realize at all, such as tail-distribution concepts. The paper does not explain how such cases are handled.
- No measure of realism after repair. The paper does not examine whether repaired candidates, with changed masks, strengths, or prompts, drift away from the natural image manifold. The reported metrics do not capture realism.
W2. The evaluation does not adequately support the claimed reliability.
- Pixel metrics favor no-op edits.
- Preservation is measured with PSNR, SSIM, and LPIPS. With mask-based inpainting editors, these metrics mostly reward leaving pixels unchanged outside the mask, and a minimal or near-no-op edit can score highly.
- The large PSNR gain (25.4→38.0 dB) therefore mainly reflects the masking design rather than edit quality.
- No human evaluation. Instruction compliance is judged only by MLLMs (GPT-5, Gemini 2.5 Pro, and Qwen3-VL-Plus), and no human evaluation is reported. This is a notable omission for a paper whose central claim is reliability.
- Missing frontier editors. There are no comparisons with modern editors such as Qwen-Image-Edit or FLUX.1 Kontext, which the paper itself mentions in the related work. The base editors (BrushNet and SDXL-Inpainting) are relatively old inpainting models.
- Unequal budgets. The benchmark baselines use published results (Sec. 4) and receive no test-time scaling. CG-TTS uses up to 8 editor calls plus GPT-4o, LISA, Grounding DINO, and MLLM verification. Moreover, the "same editor-call budget" counts only editor calls.
Questions for authors
- How does CG-TTS behave when LISA or Grounding DINO mislocalizes the target? How often does this happen on ReasonEdit, and what fraction of the remaining failures does it explain?
- How are candidates with multiple co-occurring failure types diagnosed and repaired?
- Can you report a human evaluation of instruction compliance and realism, and a comparison against frontier editors (e.g., Qwen-Image-Edit, FLUX.1 Kontext) given the same total compute, including the MLLM and grounding calls?
6 comments
Response to Reviewer 8deL (Part 2):
W1.5, Q2: Co-occurring failures
A4: Thank you for asking how Eq. (7) handles co-occurring failures. CG-TTS handles such cases by prioritizing one failure for repair while retaining all mandatory verification requirements. If a candidate remains unsuccessful after exhausting the repair budget, any fallback retains its non-PASS status. The single label in Eq. (7) identifies only the failure prioritized for the next repair action; other co-occurring failures are not discarded and remain subject to the original mandatory checks.
-
Multiple failures. CG-TTS determines the primary failure condition based on the results of individual checks: explicit failures of operation-critical checks are prioritized, followed by target-realization failures, while other failures are ranked according to their severity. The selected repair updates its generation configuration, while all original requirements remain active. The resulting candidate is then evaluated against the unchanged contract. For example, when over-editing and semantic mismatch occur simultaneously, even if repair improves preservation in the protected region, the candidate is still rejected as long as the semantic requirement remains unsatisfied.
-
Long-tail concepts. Repair provides a bounded search for a candidate satisfying the contract, using the same frozen editor. If the budget is exhausted without an all-PASS candidate, any fallback retains its non-PASS status; the acceptance criteria remain unchanged.
We clarify the distinction between repair selection and joint verification in Section 3.4 (line 277).
W1.6: Realism after repair
A1.6: Thank you for this helpful suggestion. Our additional qualitative comparisons demonstrate that repair can improve target realization and visual details, enhance photographic realism, and maintain the visual consistency of the original scene. Specifically, we have extended our evaluation beyond preservation metrics by adding human assessments of both photographic realism and instruction compliance, together with qualitative before-and-after comparisons of representative repair results.
The individual human assessment scores are reported in Section 4.1 (Tables 1 and 3), and the corresponding qualitative comparisons are provided in Appendix B.2 (Fig. 5). For example, in the sunglasses replacement case, the initial candidate exhibits excessive facial smoothing and an altered head pose. After repair, the sunglasses frames become clearer and the facial details more natural, while the subject's original pose and appearance are largely preserved. Similarly, the clothing and mask replacement cases show improved realization of the requested edits, while the wolf replacement case demonstrates better target realization while largely preserving the right cat and the surrounding scene.
W2.1: Preservation metrics and edit quality
A2.1: Thank you for your comments regarding the evaluation of editing quality. Our evaluation jointly considers content preservation and editing effectiveness through complementary metrics and controlled comparisons. Specifically:
-
Complementary evaluation. We separately assess non-target content preservation and editing effectiveness. Following SmartEdit [1], we use PSNR, SSIM, and LPIPS to measure non-target content preservation. The PSNR improvement from 25.393 to 37.964 dB reflects this aspect. Editing effectiveness is evaluated separately through instruction–crop alignment (CLIPSim), independent MLLM-based IC assessments, and the newly added human evaluation.
-
Evaluation masks and edit validity. Preservation alone is insufficient for candidate acceptance; candidates must also satisfy the required editing operations. For CG-TTS, benchmark-provided GT masks are used only for post-hoc metric computation; inference uses LISA-predicted masks. Candidate acceptance additionally requires actual image changes, sufficient change coverage, and satisfaction of operation-specific semantic requirements.
-
Controlled comparisons. Under shared localization and candidate-generation conditions, contract-guided selection provides additional gains in both CLIPSim and instruction compliance. Table 1 compares CG-TTS with CLIP Best-of-8 using the same localization and candidate pool. On ReasonEdit-U, CLIPSim improves from 21.594 to 24.075, while GPT-5/Gemini IC improves from 0.744/0.669 to 0.794/0.694.
Reference:
[1] Huang, Y., et al. SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models. CVPR, 2024.
Response to Reviewer 8deL (Part 3):
W2.2: Human evaluation.
A2.2: Thank you for highlighting the importance of human evaluation. Our human evaluation shows that CG-TTS (BrushNet) achieves the highest human composite scores on ReasonEdit, while the SDXL Inpainting variant scores highest on HumanEdit-core (Table R4). These assessments provide an independent human perspective on editing quality, complementing our MLLM-based evaluation. Specifically, we conducted human assessments on 25 Understanding and 25 Reasoning tasks from ReasonEdit, and 50 eligible tasks from HumanEdit-core. For ReasonEdit, we sampled five tasks from each of its five editing categories in the Understanding subset, and randomly sampled 25 tasks from the Reasoning subset. For HumanEdit-core, we used stratified sampling approximately proportional to the distribution of editing types. The evaluation involved 12 human raters, with each output independently assessed by two different raters. Each rater scored instruction compliance and photographic realism, and the normalized scores were averaged across the two dimensions to obtain the composite Human score. The normalized composite scores are reported in Table R4.
On ReasonEdit, CG-TTS (BrushNet) achieves 0.857/0.877 (U/R), compared with 0.623/0.510 for VIEScore Best-of-8, the strongest scalar-based selection baseline in this human evaluation. On HumanEdit-core, CG-TTS (SDXL Inpainting) achieves 0.813, compared with 0.711 for ADE-CoT and 0.682 for the SDXLInpainting baseline.
The resulting Human scores are reported in the last columns of Table 1 and Table 3. The corresponding experimental analysis is presented in Section 4.1 (lines 365 and 376), while the evaluation metrics and score computation are detailed in Section 4 (line 296) of the revised manuscript.
Table R4: Results with human evaluation.
| Method | HumanEdit-core Human ↑ | ReasonEdit (U/R) Human ↑ |
|---|---|---|
| InstructPix2Pix | 0.389 | 0.437/0.369 |
| MGIE | 0.284 | 0.304/0.386 |
| HIVE | 0.439 | 0.358/0.443 |
| MagicBrush | 0.495 | 0.250/0.487 |
| Qwen-Image-Edit | 0.732 | 0.450/0.630 |
| FLUX.1 Kontext-dev | 0.718 | 0.473/0.677 |
| SmartFreeEdit | 0.478 | 0.303/0.503 |
| BrushNet | 0.497 | 0.533/0.493 |
| SDXL Inpainting | 0.682 | 0.457/0.543 |
| ADE-CoT | 0.711 | 0.667/0.713 |
| ELECT | 0.504 | 0.400/0.437 |
| CLIP Best-of-8 | 0.451 | 0.587/0.483 |
| VIEScore Best-of-8 | 0.419 | 0.623/0.510 |
| EditScore Best-of-8 | 0.409 | 0.587/0.470 |
| CG-TTS (BrushNet) | 0.552 | 0.857/0.877 |
| Gain vs. BrushNet | +0.055 | +0.323/+0.383 |
| CG-TTS (SDXL Inpainting) | 0.813 | 0.659/0.583 |
| Gain vs. SDXL Inpainting | +0.131 | +0.203/+0.040 |
W2.3: Frontier editors
A2.3: Thank you for this helpful suggestion. On ReasonEdit, CG-TTS (BrushNet) achieves higher instruction-compliance scores than the two frontier editors. On HumanEdit-core, CG-TTS (SDXL Inpainting) achieves a higher human composite score and better image-fidelity metrics. As requested, we have expanded our comparisons to include Qwen-Image-Edit and FLUX.1 Kontext-dev on both ReasonEdit and HumanEdit-core, as reported in Table R5 and Table R6. All methods are evaluated under the same maximum budget of eight editor calls per input. For both frontier editors, we evaluate Best-of-8 results by generating eight candidates per input and selecting the final output based on CLIP scores, following the corresponding benchmark evaluation protocol. CG-TTS starts with four candidates and allocates additional editor calls to on-demand repair, subject to the same eight-call budget.
Table R5: Comparison with frontier image editors on ReasonEdit (U/R).
| Method | PSNR ↑ | LPIPS ↓ | SSIM ↑ | CLIPSim ↑ | GPT-5 IC ↑ | Gemini 2.5 Pro IC ↑ | Human ↑ |
|---|---|---|---|---|---|---|---|
| FLUX.1 Kontext-dev (Best-of-8) | 13.616/12.179 | 26.32/25.60 | 0.372/0.247 | 17.662/17.515 | 0.247/0.529 | 0.255/0.524 | 0.446/0.621 |
| Qwen-Image-Edit (Best-of-8) | 12.682/11.993 | 32.62/27.27 | 0.409/0.336 | 22.378/19.664 | 0.587/0.764 | 0.681/0.592 | 0.487/0.630 |
| CG-TTS (BrushNet) | 37.964/36.562 | 10.59/11.10 | 0.986/0.986 | 24.075/20.829 | 0.794/0.869 | 0.694/0.781 | 0.857/0.877 |
| CG-TTS (SDXL Inpainting) | 40.882/42.376 | 8.42/4.06 | 0.989/0.993 | 22.579/19.899 | 0.525/0.333 | 0.419/0.350 | 0.659/0.583 |
Table R6: Comparison with frontier image editors on HumanEdit-core.
| Method | L1 ↓ | L2 ↓ | CLIP-I ↑ | DINO ↑ | CLIP-T ↑ | GPT-5 IC ↑ | Gemini 2.5 Pro IC ↑ | Human ↑ |
|---|---|---|---|---|---|---|---|---|
| FLUX.1 Kontext-dev (Best-of-8) | 0.1231 | 0.0419 | 0.8954 | 0.7552 | 0.1935 | 0.663 | 0.403 | 0.618 |
| Qwen-Image-Edit (Best-of-8) | 0.1359 | 0.0474 | 0.9232 | 0.8300 | 0.1800 | 0.801 | 0.697 | 0.632 |
| CG-TTS (BrushNet) | 0.0467 | 0.0169 | 0.8904 | 0.8105 | 0.2159 | 0.442 | 0.233 | 0.552 |
| CG-TTS (SDXL Inpainting) | 0.0395 | 0.0134 | 0.9412 | 0.9034 | 0.2157 | 0.615 | 0.489 | 0.813 |
Response to Reviewer 8deL (Part 1):
Thank you for your careful review and for recognizing the value of requirement-level verification and budget-aware repair. We also appreciate your suggestions to clarify the methodological contribution and scaling behavior, and to strengthen the failure analysis and evaluation. We provide the corresponding clarifications, analyses, and additional experimental results below. All revisions and additions to the main text are highlighted in red in the revised manuscript for ease of review.
W1.1: Unclear "scaling".
A1: Thank you for requesting clearer evidence of test-time scaling. CG-TTS scales test-time computation through candidate exploration and verification-guided repair, using verification feedback to determine whether additional generation is needed and how the generation configuration should be updated. The contract and acceptance criteria remain fixed.
Specifically, Figure 3 and Section 4.3 compare budget settings using the same editor and auxiliary components. With four initial candidates, allowing one conditional repair round increases the maximum editor-call budget from 4 to 8 and average usage from 4 to 4.98 on ReasonEdit-U. CLIPSim improves from 20.737 to 24.075, and PSNR from 37.587 to 37.964 dB. This comparison quantifies the incremental benefit of additional inference after the external tools are already in place.
W1.2–W1.3: Methodological novelty
A2: Thank you for asking us to clarify the methodological contribution. CG-TTS introduces a contract-defined inference policy: an explicit semantic–spatial contract specifies admissible information reuse, candidate acceptance, and failure-conditioned generation. The contributions are:
Contract-conditioned SMR. SMR builds on attention and feature reuse explored in Prompt-to-Prompt [1], MasaCtrl [2], and PnP [3]. Its contribution is using the editing operation and spatial contract to determine admissible memory sources, update regions, and reinforcement or suppression terms. For replacement, SMR can combine target-positive routing with suppression of source-exclusive features; protected-region features remain available as memory but receive no updates (Table 8). Removing SMR reduces CLIPSim from 24.075 to 22.118 and PSNR from 37.964 to 35.004 dB, quantifying its contribution within CG-TTS (Table 4).
Contract-defined acceptance and repair. The contract establishes explicit dependencies between generation constraints, acceptance decisions, and subsequent interventions. The same contract requires every mandatory check to pass before soft ranking. For example, a failed target-presence check cannot be compensated by a high preservation score. If no candidate passes, diagnosed failures guide repair while the original acceptance conditions remain fixed. With the same candidate pool, this acceptance policy reduces constraint violations from 49.6% to 21.4% (Table 2).
We make these methodological distinctions explicit in the contribution statement and Sections 3.3–3.4.
References:
[1] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompt Image Editing with Cross-Attention Control. International Conference on Learning Representations (ICLR), 2023.
[2] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 22560–22570.
[3] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1921–1930.