acceptodds
Under review as a conference paper at ICLR 2027

Learning Minimal Sufficient Visual Intervention for Instruction-Based Image Editing

Abstract

Recent advances in instruction-based image editing have enabled high-quality edits from source images and language instructions. However, the visual intervention needed to translate user intent into executable editing conditions typically remains implicit, leaving the intended change boundary underspecified and potentially leading to incomplete edits or unintended modifications to unrelated content. In this work, we formulate instruction-based image editing as learning a minimal sufficient visual intervention (MSVI) that fulfills the instruction while minimizing changes to instruction-invariant visual content. Specifically, we first use a multimodal large language model as an edit planner to predict a structured edit specification encoding the intervention regime, visual transition, semantic actions, and spatial layout. Based on the predicted specification, a deterministic compiler then invokes SAM 3 only when the localized grounding is required. The resulting outputs form the intervention condition supplied to a flow-matching editor. To supervise the planner and the editor, we further construct MSVI-13K with annotated structured edit specifications. The annotations provide prediction targets for the planner and determine the intervention conditions used to train the editor with conditional flow matching and a source-copy suppression loss. Extensive experimental results on MSVI-13K and two external benchmarks show that our method matches or outperforms state-of-the-art approaches, supporting explicit visual intervention learning for balancing instruction fulfillment and content preservation. Code is available at https://anonymous.4open.science/r/MSVI/.

Primary area:
generative models
Keywords:
instruction-based image editing, diffusion models, multimodal large language models, visual reasoning, controllable image generation, conditional flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6hDkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a key pain point in instruction‑guided image editing: existing methods often keep visual intervention boundaries implicit, leading to incomplete edits or undesired modifications to unchanged regions. The authors propose Minimal Sufficient Visual Intervention (MSVI), a three‑stage pipeline: (1) an VLM planner outputs a Structured Edit Specification (SES) with planning, perceiving, action and layout fields; (2) a regime‑adaptive compiler produces explicit visual intervention conditions Z, conditionally invoking SAM3 for spatial grounding; (3) a flow‑matching MMDiT editor generates edited images conditioned on Z. The authors build MSVI‑13K dataset with structured edit annotations. Extensive experiments on MSVI‑13K, Reason‑Edit and PIE‑Bench demonstrate competitive performance against strong baselines. Ablation studies validate each key component.

Critical strengths

S1: This work formalizes instruction‑based image editing problem as learning minimal sufficient visual intervention(MSVI). The proposed explicit paradigm, to some extent, mitigates two common problems: incomplete editing and unintended content corruption. S2: The SES structured representation, regime‑adaptive compilation with SAM3 grounding, and source‑copy suppression loss form a credible and coherent system. The MSVI‑13K dataset provides structured edit‑specification annotations, which can benefit follow‑up research for reasoning‑driven image editing. S3: Quantitative comparisons, qualitative visualization, cross‑benchmark generalization test and thorough ablation are solid. The paper also analyzes some representative failure cases and discusses limitations.

Critical weaknesses

W1: Planner and flow‑based editor are trained separately(not end-to-end). Inference errors from SES prediction and SAM3 will propagate to final output, as acknowledged by authors. There is no closed‑loop feedback mechanism to correct wrong intermediate outputs. W2: The pipeline introduces MLLM planner, SAM3 segmenter, VAE and MMDiT flow executor into the solution. It requires multiple external models and multi‑step inference, raising deployment overhead compared to end‑to‑end single‑model competitors. W3: The paper only evaluates open‑source academic baselines, without comparing against successful closed‑source commercial systems such as Seedream, GPT‑Image‑2.0 and Gemini. It remains unclear how the proposed MSVI stacks up against state‑of‑the‑art commercial editors under the same evaluation protocols.

Questions for authors

Q1: When SES prediction makes wrong regime prediction (e.g. predicting FOCAL for a HOLISTIC sample), how severe is the downstream generation degradation? Have you quantified this effect? Q2: Could you elaborate why source‑copy suppression loss (SCSL) brings such huge gain for edit sufficiency? Is there alternative simpler loss design to achieve similar effect? Q3: For real‑world deployment, do you have thoughts to simplify the multi‑component pipeline, e.g. distill the whole MSVI logic into a single end‑to‑end generative model?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer 6hDk (part 1)

We thank the reviewer for recognizing how structured edit specifications, regime-adaptive grounding, and source-copy suppression work together, and for noting the value of the cross-benchmark evaluation. We also appreciate the reviewer pointing out the risk of error propagation, the deployment overhead, and the lack of commercial baselines. We explain why the planner and editor are trained separately, report editing scores for correct and incorrect regime predictions, and compare inference latency and editing quality with other editors.

1. [W1]: Why are the planner and editor trained separately, and how can errors propagate without closed-loop feedback?

A: We agree that the reliability of intermediate predictions and the computational cost of the complete system are important for assessing practical image editing. MSVI provides fully automated inference from a source image and instruction to the edited output, while deliberately adopting stage-wise optimization (main paper Section 3.4). We first train the planner with direct SES supervision, and then train the executor using conditions compiled from annotated SES, with the planner and VAE frozen. This gives the executor a fixed supervised conditioning interface and prevents executor optimization from altering the learned specification predictor. Decoupled optimization of multimodal language and diffusion components also has precedent in LLMGA (Xia et al., 2024) and WeGen (Huang et al., 2025), although their specific training schedules differ. Our design is intended to reduce optimization coupling; it does not guarantee error-free SES prediction.

[1] Xia, Bin, et al. "Llmga: Multimodal large language model based generation assistant." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Huang, Zhipeng, et al. "Wegen: A unified model for interactive multimodal generation as we chat." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

2. [Q1, W1]: How much does editing performance degrade when the planner predicts the wrong regime?

A: To further investigate the effect of regime prediction errors, we divided the test samples into two subsets according to whether their predicted editing regimes matched the ground-truth labels. The corresponding downstream editing performance is reported in Table C-1.

Evaluation subsetSamples (%)ER-IF ↑InvPres ↑
Correct regime predictions83.41.8070.879
Incorrect regime predictions16.61.3010.374

Table C-1. Downstream editing performance grouped by edit-type prediction correctness on the MSVI-13K test set.

As shown in Table C-1, incorrect regime predictions are associated with substantial performance degradation, reducing ER-IF from 1.807 to 1.301 and InvPres from 0.879 to 0.374. This highlights the importance of accurate regime selection for both instruction fulfillment and content preservation. Nevertheless, with 83.4% of predictions correctly classified, the planner achieves strong performance on the majority of test cases. These findings provide a more comprehensive assessment of the impact of planning errors on downstream editing.

AuthorsOpenReview ↗

Reply to Reviewer 6hDk (part 2)

3. [Q3, W2]: What is MSVI’s end-to-end inference cost, and could the pipeline be distilled into a single model?

A: We appreciate the reviewer’s practical concern regarding deployment overhead and the suggestion to further simplify the pipeline. Component count alone does not necessarily determine deployment efficiency. We therefore evaluated the complete MSVI pipeline against BAGEL and Qwen-Image-Edit-2509, measuring end-to-end latency on an NVIDIA RTX PRO 6000 GPU at an image resolution of 512 × 512 pixels and a batch size of 4 for all methods. For MSVI, the reported latency covers the complete pipeline, including planning, intervention compilation, conditional grounding, and the 40-step flow-generation process.

MethodInference settingMean latency (s/image) ↓
BAGEL50 steps; thinking off32.2
Qwen-Image-Edit-250940 inference steps27.2
MSVI (ours)40 flow steps12.3

Table C-2. End-to-end inference latency of MSVI and recent unified image-editing models. All methods are evaluated under the same hardware, image resolution, and batch size. Each value reports the mean latency over the complete inference pipeline.

As shown in Table C-2, MSVI achieves substantially lower end-to-end latency than BAGEL (12.3 vs. 32.2 s/image) and Qwen-Image-Edit-2509 (12.3 vs. 27.2 s/image), despite explicitly introducing planning and grounding stages. This indicates that the modular structure of MSVI does not necessarily translate into higher practical inference overhead.

We also appreciate the reviewer’s suggestion of distilling MSVI into a unified model. Combining pretrained language and generative models is an established approach for bridging multimodal reasoning and visual generation, as explored in LLMGA and WeGen. A natural future direction is to distill the MSVI pipeline into a single generative model, using SES and the compiled spatial supports as intermediate supervision in addition to the final editing targets. Such a student could internalize planning and grounding while reducing separate model calls at inference, further simplifying deployment without discarding the explicit intervention supervision introduced by MSVI.

4. [Q2]: Why does SCSL improve edit sufficiency, and could a simpler loss achieve a similar effect?

A: We thank the reviewer for this insightful question and the suggestion to explore simpler loss designs. For localized edits, the requested changes may occupy only a small portion of the image. With a finite training budget, a global objective can underemphasize these changes, resulting in conservative, source-like outputs that under-execute the instruction. SCSL counteracts this tendency by applying area-normalized target matching within the intervention support, while retaining full-image flow supervision. This increases the training emphasis on completing the requested transition. Table 3(c) of the main paper is consistent with this explanation: removing SCSL yields stronger source preservation but reduces ER-IF from 1.630 to 0.503. Regarding simpler alternatives, under our flow parameterization, SCSL is equivalent to a support-normalized velocity-matching term with a σ2\sigma^2 time weight (Eqs. (7) and (9)). A natural simplification is to use a time-independent weight on the regional flow-matching term. We appreciate this suggestion and will explore whether such alternatives can achieve a comparable balance between edit sufficiency and preservation in future work.

AuthorsOpenReview ↗

Reply to Reviewer 6hDk (part 3)

5. [W3]: How does MSVI compare with Seedream, GPT-Image-2, and Gemini under the same evaluation protocol?

A: We appreciate the reviewer’s suggestion to include commercial image editors. We therefore extended the evaluation to Seedream 5.0 Lite, GPT-Image-2, and Gemini (Nano Banana 2), using the same source images, editing instructions, and quality-evaluation protocol.

The comparison reports both editing quality and observed end-to-end response latency. For commercial APIs, timing covers request submission through retrieval of the edited image and therefore includes communication and service overhead. We interpret these measurements as user-facing response times rather than hardware-matched estimates of model computation. Model versions, generation settings, and evaluation dates were kept fixed throughout the evaluation.

MethodDeploymentMean response latency (s/image) ↓Instruction success rate (%) ↑ER-IF ↑
GPT-Image-2API45.092%1.799
Gemini (Nano Banana 2)API138.976%1.522
Seedream 5.0 LiteAPI33.677%1.544
MSVI (ours)Local12.384%1.630

Table C-3. Comparison with closed-source commercial image editors in terms of user-facing response latency and instruction-following performance.

As shown in Table C-3, MSVI achieves the lowest response latency (12.3 s/image) while outperforming Gemini and Seedream 5.0 Lite in both instruction success rate and ER-IF. Although GPT-Image-2 achieves the strongest editing accuracy, MSVI provides a competitive instruction-following performance with substantially lower end-to-end latency, demonstrating a favorable quality–efficiency trade-off.


If the reviewer has any further questions, we would be happy to discuss them.

Reviewer_wrE5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces minimal sufficient visual intervention into instruction-based image editing (IIE), aiming to fulfill the instruction while mitigating incomplete edits and unintended modifications. Specifically, the authors first utilize a multimodal large language model to generate a structured edit specification, and a deterministic compiler then invokes SAM 3 only when localized grounding is required. The resulting outputs form the intervention condition supplied to a flow-matching editor. To supervise the MLLM-based planner and the flow-matching-based editor, the authors also curated MSVI-13K. Extensive experimental results show that their method outperforms SOTA baselines.

Critical strengths
  1. This work reformulates the conventional instruction-based image editing (IIE) task as learning a minimal sufficient visual intervention, focusing more on local editing.

  2. The authors utilize an MLLM to predict a structured edit specification comprising four dimensions, rather than asking the MLLM to reason about concise, expressive instructions and directly provide explicit visual guidance (e.g., MGIE).

  3. The authors curate a new dataset for planner and executor fine-tuning, and the experimental results show its potential to improve existing methods.

  4. The authors conduct extensive experiments to demonstrate the effectiveness of the proposed framework from diverse perspectives.

Critical weaknesses
  1. The MLLM-based planner is evaluated only with Qwen3.5-4B. Could a more powerful MLLM generate more accurate structured edit specifications and achieve comparable performance without fine-tuning? An analysis across different MLLM architectures and sizes would help clarify the necessity of planner fine-tuning.

  2. The failure cases in Figure S5 involve ambiguous object identification and occluded instance boundaries. However, it is unclear whether these failures originate from the MLLM-based planner or subsequent grounding and editing stages. If they originate from the planner, could a stronger MLLM alleviate these issues? This also raises concerns about how strongly the framework’s performance depends on the underlying MLLM.

  3. Lacks comparisons with the latest image editing models, such as GPT-Image-2 and BAGEL[1], etc. Furthermore, how does the proposed framework perform on tasks requiring substantial reasoning, such as those in EditWorld [2] and Reason50K [3]?

  • [1] Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025.
  • [2] Yang, L., Zeng, B., Liu, J., Li, H., Xu, M., Zhang, W., and Yan, S. Editworld: Simulating world dynamics for instruction-following image editing. arXiv preprint arXiv:2405.14785, 2024.
  • [3] He, Q., Chen, X., Wang, C., Pan, Y., et al. Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoning. Proceedings of the 43rd International Conference on Machine Learning, 2026
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer wrE5 (part 1)

We thank the reviewer for recognizing the value of the four-field edit specification and the potential of MSVI-13K to improve other editors. We also appreciate the reviewer raising questions about planner fine-tuning, the causes of the failures in Figure S5 of the Supplementary Material, and the coverage of the evaluation. We compare fine-tuned and zero-shot planners with the same compiler and editor, discuss which stages could account for the two failures, and report comparisons with BAGEL and GPT-Image-2 on EditWorld and Reason50K.

1. [W1]: Can a stronger zero-shot MLLM match the fine-tuned Qwen3.5-4B planner?

A: We appreciate the reviewer’s suggestion to examine the effect of planner capacity. A more capable MLLM can potentially improve visual reasoning and structured edit specification quality. Our choice of Qwen3.5-4B is motivated by the trade-off between planning quality and inference overhead. Task-specific fine-tuning enables a compact planner to better follow the SES schema and the instruction-following requirements of image editing while maintaining low planning latency.

To examine this trade-off, we compared our fine-tuned Qwen3.5-4B planner with its zero-shot counterpart, a substantially larger Qwen planner, and GPT-5.6 Luna through API access. All planners received the same source images and instructions, followed the same SES schema, and shared the same fixed compiler and editor. We evaluated SES validity, regime accuracy, downstream instruction fulfillment and preservation, and planning latency.

PlannerAdaptationSES validity (%) ↑Regime Acc. (%) ↑ER-IF ↑BG-LPIPS ↓Planning latency (s/sample) ↓
Qwen3.5-4BZero-shot76.2%77.9%1.0830.2444.54
Qwen3.5-4B (ours)SES fine-tuning89.6%83.4%1.6300.1504.53
Qwen3-27BZero-shot86.5%87.7%1.4820.18343.2
GPT-5.6 luna (Use API)Zero-shot94.3%93.6%1.7060.148around 3min

Table B-1. Comparison of planner capacity and task-specific adaptation under a fixed compiler and editor. Note that GPT-5.6 Luna latency includes API communication and service overhead and is therefore not directly hardware-comparable to the locally evaluated Qwen models.

As shown in Table B-1, SES fine-tuning substantially improves the compact Qwen3.5-4B planner over its zero-shot counterpart and even outperforms the much larger Qwen3-27B in downstream editing quality, while retaining an order-of-magnitude lower latency. Although GPT-5.6 Luna achieves slightly higher ER-IF and lower BG-LPIPS, it incurs substantially higher response latency, supporting our choice of a task-adapted 4B planner as a favorable quality–efficiency trade-off.

2. [W2]: Which stages could explain the failures in Figure S5 of the Supplementary Material, and could a stronger planner help?

A: The failure cases arise from both the planner and the downstream grounding/generation stages. As discussed in Supplementary Material Section S6, ambiguous scenes with multiple visually similar instances can cause the planner to produce an inaccurate referring expression or spatial specification, while complex occlusions and uncertain object boundaries can still lead to grounding or synthesis failures even when the high-level plan is reasonable.

A stronger MLLM can improve the planner side of this pipeline. This is also supported by the additional results in Table B-1, where larger or more capable planners generally produce stronger SES predictions and downstream editing performance. Importantly, MSVI is not tied to Qwen3.5-4B; the planner can be replaced by a stronger MLLM through the same SES interface.

However, increasing planner capacity does not directly resolve failures caused by segmentation uncertainty, occluded boundaries, or generation under correct spatial conditions. Our use of a compact fine-tuned planner therefore reflects a quality–latency trade-off rather than an architectural limitation. Exploring substantially larger planners is possible within MSVI, but is not the primary focus of this work and, as shown in Table B-1, can introduce considerably higher inference latency.

AuthorsOpenReview ↗

Reply to Reviewer wrE5 (part 2)

3. [W3]: How does MSVI compare with BAGEL and GPT-Image-2 on EditWorld and Reason50K?

A: We appreciate the reviewer’s suggestion to broaden the comparison to recent image editing systems and more reasoning-intensive benchmarks. We therefore added BAGEL [1] and GPT-Image-2 to the evaluation using the same inputs and evaluation protocols.

For GPT-Image-2, its closed-source API does not expose model internals or training configurations, making matched-training and architectural comparisons infeasible. We therefore use it primarily as a reference for output quality rather than as a hardware-matched efficiency baseline. This distinction is also consistent with the latency analysis in Table B-1: API-based large models can provide stronger performance, but may introduce substantially higher user-facing latency than compact locally deployed models.

Beyond the existing Reason-Edit evaluation in Table 2 of the main paper, we further evaluated the models on EditWorld [2] and the held-out Reason50K validation set [3]. EditWorld emphasizes editing under world dynamics, while Reason50K covers physical, temporal, causal, and story reasoning. All models were evaluated using fixed checkpoints without benchmark-specific fine-tuning and following the corresponding benchmark protocols. 0.172 0.781 0.482 7.85

MethodEditWorld: CLIP† ↑EditWorld: MLLM† ↑Reason50K: CLIP ↑Reason50K: MLLM ↑Reason50K: Ins-Align ↑Reason50K: IP ↑
BAGEL0.22500.87730.1720.7810.4827.85
GPT-Image-20.27300.94830.1690.8810.6948.64
MSVI (ours)0.25390.93720.1670.8670.6388.38

Table B-3. Comparison with recent open- and closed-source image editing models on EditWorld and Reason50K. All methods are evaluated using fixed checkpoints without benchmark-specific fine-tuning and under the original evaluation protocols of the respective benchmarks.

As shown in Table B-3, MSVI outperforms BAGEL on five of the six reported metrics, while achieving a competitive score and remains competitive with the closed-source GPT-Image-2. In particular, MSVI approaches GPT-Image-2 closely on EditWorld MLLM (0.9372 vs. 0.9483) and Reason50K CLIP (0.167 vs. 0.169), while maintaining the substantially more efficient compact-planner design demonstrated in Table B-1. These results further support the generalization of MSVI to reasoning-intensive editing beyond the datasets used for training.

[1] Deng, Chaorui, et al. "Emerging properties in unified multimodal pretraining." arXiv preprint arXiv:2505.14683 (2025).

[2] Zeng, Bohan, et al. "Editworld: Simulating world dynamics for instruction-following image editing." Proceedings of the 33rd ACM International Conference on Multimedia. 2025.

[3] He, Qingdong, et al. "Reasoning to edit: Hypothetical instruction-based image editing with visual reasoning." arXiv preprint arXiv:2507.01908 (2025).


If the reviewer has any further questions, we would be happy to discuss them.

Reviewer_wrE5OpenReview ↗

All my concerns have been addressed.

Thanks to the authors' rebuttal. As all my concerns have been addressed, I would like to slightly raise my initial score.

AuthorsOpenReview ↗

Reply to Reviewer wrE5

We thank the reviewer wrE5 for the careful re-reading and for a discussion that materially improved the paper.

Reviewer_FkRbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at a common failure of instruction-based image editors. Given an instruction like "change the cat not in the mirror to a red panda", an editor may change too little (an incomplete edit) or too much (it also alters things that should stay the same). The authors frame the goal as finding a "minimal sufficient visual intervention": a change that does what the instruction asks and nothing more.

The method has three stages. First, a fine-tuned multimodal LLM (Qwen3.5-4B) reads the image and instruction and writes a structured plan with four parts: the edit type (local, whole-image, or both), a description of the source and target states, a list of editing steps, and a layout with bounding boxes. Second, a fixed rule-based "compiler" turns this plan into conditions for the image model. For local edits it calls SAM 3 to get a mask and draws a padded box around it. It also fills a mask from the layout boxes and encodes the steps as text. Third, an SD3.5-Medium editor, fine-tuned with LoRA, generates the edited image from these conditions. Training adds a masked loss that pushes the predicted clean image toward the target inside the edit region, to stop the model from just copying the source.

To train the planner and editor, the authors build MSVI-13K. It has about 13K triplets taken from AnyEdit, CompBench, and SmartEdit data. The structured plans are generated by Qwen3.5-9B, which sees both the source and the target image. The method is evaluated on the MSVI-13K test split and on Reason-Edit and PIE-Bench. It is compared with twelve editors and reports the best or near-best scores on most metrics. Ablations in Table 3 remove plan fields, compiler parts, and the extra loss.

Critical strengths

S1. A clear and useful problem framing. The paper separates two failure modes that are often mixed together: edits that do too little and edits that do too much. It also points out that "minimal" does not mean "small". A style change over the whole image can still be minimal if the instruction asks for it. The three edit types (local, whole-image, combined) give a simple way to decide when a mask should be used and when it should not. This is a practical idea that other editing systems could adopt.

S2. A reasonably broad experimental setup and a useful dataset. The comparison covers twelve editors from four families, including recent strong models such as Qwen-Image-Edit, Step1X-Edit, OmniGen2, GoT, and RePlan. The ablation in Table 3 includes a length-matched unstructured plan, which is a good control for the question "is it the structure or just the extra text?". The MSVI-13K dataset with structured plans, if released, could help other planning-based editors. The paper shows this in Section 4.2 by fine-tuning SmartEdit and GoT on it.

Critical weaknesses

W1. The main evaluation is not a fair or independent test of the method. This is my biggest concern.

  • In-domain versus out-of-domain. MSVI is trained on the MSVI-13K training split and tested on its test split, which comes from the same sources and the same annotation pipeline. The baselines in Table 1 are run with "their official inference settings" (Section 4, Baselines), so for them this is an unseen distribution. A large part of the gap in Table 1 may come from in-domain fine-tuning rather than from the structured intervention. The fair comparison is a baseline fine-tuned on the same training split with the same backbone. The closest control in the paper is "Direct instruction conditioning" in Table 3(a), but it is not clear whether it is trained with the same data, steps, and loss.
  • The metrics are tied to the method's own labels. Int-F1 and ETP are computed against the "annotated intervention region" (Eq. 11). These regions come from the same Qwen3.5-9B pipeline that produced the planner's training targets (Appendix S1). So MSVI is trained to predict exactly the region it is later scored against, while baselines are not. The remaining metrics (EditReward-IF and EditReward-VQ) come from a learned reward model. There is no human study. This makes it hard to trust the headline claims such as the average gain in instruction fulfillment stated at the end of Section 1.
  • Unclear protocol for Table 2. Section 4.1 says Table 2 is obtained "by training all models on MSVI-13K". The Baselines paragraph says official inference settings are used. Section 4.2 then fine-tunes SmartEdit and GoT on MSVI-13K as a separate experiment, which suggests the Table 2 baselines were not fine-tuned. Which is it? Also, MSVI-13K includes SmartEdit's synthetic data, and Reason-Edit is SmartEdit's benchmark. The paper should confirm there is no overlap in images or instructions.
  • No variance is reported. All tables give single numbers. Several margins over the best baseline in Table 2 (for example BG-LPIPS and CLIP-Edit on PIE-Bench) are very small, and MSVI is not best on some of them. Without seeds or confidence intervals, it is hard to say MSVI "provides the stronger joint" result.

W2. The "minimal sufficient" formulation is not what the method actually optimizes, and the ablations raise more questions than they answer.

  • Minimality is defined in Section 1, but no part of the method enforces it. Eq. 10 is a planner cross-entropy, a standard flow-matching loss, and a masked loss that only pulls the output toward the target inside the edit region (Eq. 9). Nothing penalizes changes outside the region, and nothing checks that the predicted intervention is the least change among sufficient ones. In practice "minimal" means "whatever the Qwen3.5-9B annotator wrote after seeing the target image". The framing in the title and Section 1 is therefore stronger than what the method delivers. The paper also cites Soatto and Chiuso for sufficiency, but no formal link to that notion is made.
  • The source-copy suppression loss seems to drive most of the gain. In Table 3(c), removing it drops instruction fulfillment below the "direct instruction conditioning" variant and below most baselines. So the plain editor barely edits, and much of the improvement comes from this masked loss. This loss is essentially a masked reconstruction loss on the predicted clean latent, which is a known tool. It is also cited to McDougall et al. (2023), a paper on an attention head in language models that is not about diffusion training. The citation looks wrong.
  • Some ablation results are not explained. "w/o focal-support expansion" in Table 3(b) causes a very large collapse in InvPres and BG-LPIPS, worse than every baseline in Table 1, while Section 4.3 only says it "largely retains ER-IF". Removing any single plan field in Table 3(a) costs about a third of the instruction-fulfillment score, which is surprising for fields such as Perceiving. It is not stated whether each ablation retrains the planner, the editor, or both.
  • Train/test mismatch for the editor. Section 3.4 says the editor is trained on ground-truth plans but tested on predicted plans. There is no result with oracle plans versus predicted plans, so the cost of planner errors is unknown. The planner evaluation in Table S1 uses CLIP text similarity even for the three-way edit type, where plain accuracy would be clearer.
  • Novelty relative to prior work is modest. Using an MLLM to plan an edit and then grounding it with boxes or masks before a diffusion editor is already done by GoT, RePlan, X-Planner, and SmartEdit, which the paper cites. Other well-known related work, such as LLM-grounded Diffusion (Lian et al.) for LLM-predicted layouts and BrushEdit (Li et al.) for MLLM-guided mask-based editing, is not discussed. The paper should state more precisely what is new beyond the three-type switch and the specific plan schema.
Questions for authors

Clarifications:

Q1. Fair training comparison. Can you report at least one strong baseline (for example the same SD3.5-Medium backbone, or Step1X-Edit or OmniGen2) fine-tuned on the MSVI-13K training split with the same budget, and then evaluated on the same test split? If MSVI still clearly beats it on instruction fulfillment and preservation, my main concern would be resolved.

Q2. Independent evaluation. Can you add a human preference study, or at least a metric that does not depend on your own annotated intervention regions? How are the annotated regions for Int-F1 and ETP produced exactly, and were the test-set labels checked by people?

Q3. Table 2 protocol. Were the baselines in Table 2 fine-tuned on MSVI-13K or run with official checkpoints? Is there any overlap between SmartEdit's synthetic data in MSVI-13K and the Reason-Edit test set?

Q4. Minimality. Is there any part of the training objective or the compiler that penalizes changes outside the predicted region? If not, how should the reader understand the "minimal" in the title?

Q5. Oracle versus predicted plans. What are the scores when the editor uses the ground-truth plans on the test set? This would show how much is lost from planner errors. Please also report plain accuracy for the edit-type field.

Q6. Ablation details. For each row of Table 3, which modules were retrained? Why does removing the focal-support expansion cause such a large drop in preservation? Please also report variance over several seeds for Tables 1 to 3.

Minor:

Q7. The source-copy suppression loss is cited to McDougall et al. (2023), which is about an attention head in language models. Please check this citation. Figure 1's caption repeats the sentence about dashed boxes.

Q8. The parameter count of 6.5B in Table 1 seems to cover the planner and the editor. Does it include SAM 3 and the text encoders? Please also report inference time compared with the baselines, since MSVI runs an MLLM, SAM 3, and 40 diffusion steps.

Q9. Will MSVI-13K with its plan annotations, the trained planner, and the editor weights be released? The anonymous code link is given, but the paper does not say whether the data is included.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer FkRb (part 1)

We thank the reviewer for highlighting the distinction between incomplete edits and unintended changes, and for recognizing the value of the length-matched planning control. We also appreciate the reviewer raising concerns about training fairness, evaluation independence, and the meaning of minimality. We clarify the baseline training and checkpoint protocols, report a blinded human preference study, and explain how the training objective encourages content preservation.

1. [Q1, W1]: Were the baselines fine-tuned on MSVI-13K, and does MSVI improve performance over a baseline trained under the same conditions?

A: All baselines in Table 1 of the main paper, including Step1X-Edit and OmniGen2, were fine-tuned on the MSVI-13K training split before evaluation on the same test split. “Official inference settings” refers to the inference configurations used with these fine-tuned checkpoints. The baselines therefore also benefit from in-domain adaptation.

Table 3(a) of the main paper already includes a direct-conditioning control using the same SD3.5-Medium executor backbone, training data, number of optimization steps, and executor objective, including both flow matching and SCSL, as MSVI. Under these matched settings, MSVI improves instruction fulfillment (ER-IF: 0.682 to 1.630) and invariant preservation (InvPres: 0.672 to 0.896), while reducing BG-LPIPS from 0.179 to 0.150. These results support the benefit of structured intervention beyond dataset adaptation or the inclusion of SCSL alone. Table 3(c) of the main paper separately evaluates the contribution of SCSL by removing it from MSVI. We will explicitly document these training settings and the objective used for each ablation in main paper Section 4.

2. [Q2, W1]: Does an independent human evaluation support the results, and how were the reference intervention regions generated and checked?

A: We appreciate the reviewer’s concern regarding evaluation independence and annotation quality. To provide an assessment independent of our intervention annotations and learned reward metrics, we conducted a blinded human preference study with 50 participants.

Human evaluation. We compared MSVI with Step1X-Edit and Qwen-Image-Edit-2509 on the same held-out inputs. Each trial displayed only the source image, editing instruction, and two anonymized outputs, with randomized left/right placement. Participants independently assessed instruction fulfillment, preservation of instruction-invariant content, and visual quality, with ties allowed. The study included 100 randomly seleted examples.

ComparisonInstruction fulfillment: W/T/L (%)Invariant-content preservation: W/T/L (%)Visual quality: W/T/L (%)
MSVI vs. Step1X-Edit54%/22%/24%76% / 14% / 10%76% / 6% / 18%
MSVI vs. Qwen-Image-Edit-250948% / 8% / 44%88% / 2% / 10%68% / 8% / 24%

Table A-1. Human preference study with 50 participants. Each entry reports the percentages of MSVI wins / ties / loses.

As shown in Table A-1, MSVI achieves preservation win rates of 76% against Step1X-Edit and 88% against Qwen-Image-Edit-2509, while receiving more wins than losses for instruction fulfillment against both baselines, providing independent human evidence for its balance between edit sufficiency and content preservation.

Annotation generation and quality control. As detailed in Supplementary Material Section S1, the annotations were constructed through a two-stage pipeline. First, Qwen3.5-9B identified visual entities and their locations from paired source and target images. Second, the model generated structured edit specifications using these detections, the image pair, and the editing instruction. The resulting Layout annotations specify edit-relevant entities and spatial intervention information, which provide the basis for the reference intervention regions used by Int-F1 and ETP. The generated SES annotations were reviewed by the authors for agreement with the source–target image pair and instruction, cross-field consistency, and unambiguous grounding, as described in Supplementary Material Section S1. While Int-F1 and ETP depend on annotated intervention regions, the additional human evaluation assesses editing outcomes independently of these annotations, providing complementary evidence beyond annotation-dependent metrics.

AuthorsOpenReview ↗

Reply to Reviewer FkRb (part 2)

3. [Q3, W1]: Which checkpoints were used in Table 2 of the main paper, and could the SmartEdit training data overlap with the Reason-Edit test set?

A: We thank the reviewer for highlighting the need to clarify the evaluation protocol and data provenance.

Evaluation protocol. Table 2 of the main paper evaluates MSVI, trained on MSVI-13K, against baselines using their officially released checkpoints, without additional fine-tuning on Reason-Edit or PIE-Bench. This differs from Table 1 of the main paper, where all baselines were fine-tuned on MSVI-13K for in-domain comparison. “Official inference settings” refers to the inference configuration and should be distinguished from the training history of the evaluated checkpoints. Figure 4 of the main paper separately compares SmartEdit and GoT before and after adaptation on MSVI-13K. We will explicitly document the checkpoint used for each baseline in Table 2 of the main paper and its relationship to the models evaluated in Table 1 of the main paper and Figure 4 of the main paper.

Data separation. The 328 synthetic triplets contributed by SmartEdit to MSVI-13K originate from the Reason-Edit training split, rather than its test split used in Table 2 of the main paper. Therefore, these samples are drawn from the training partition and do not directly include Reason-Edit test examples. We will clarify their provenance and split identifiers and verify image- and instruction-level overlap to make the data separation explicit.

Additional reasoning evaluation. To further assess transfer beyond the data sources used to construct MSVI-13K, we evaluated MSVI on EditWorld and Reason50K validation set. Reason50K covers physical, temporal, causal, and story reasoning. We used fixed checkpoints without additional fine-tuning on either benchmark and followed each benchmark’s original evaluation metrics. The results in Table A-2 provide complementary evidence of generalization to reasoning-intensive editing.

MethodEditWorld: CLIP ↑EditWorld: MLLM ↑Reason50K: CLIP ↑Reason50K: MLLM ↑Reason50K: Ins-Align ↑Reason50K: IP ↑
MagicBrush0.24320.84550.1750.7900.4417.52
MGIE0.23500.81730.1670.7540.4067.63
Qwen-Image-Edit-25090.24990.92740.1760.7820.4957.92
SmartEdit0.23870.90260.1800.8020.5328.02
BAGEL0.22500.87730.1720.7810.4827.85
MSVI (ours)0.25390.93720.1670.8670.6388.38

Table A-2. Evaluation on EditWorld and the held-out Reason50K validation set using fixed checkpoints without benchmark-specific fine-tuning.

4. [Q4, W2]: What does “minimal” mean, and how do the training objective and compiler address changes outside the edit region?

A: We thank the reviewer for highlighting the need to clarify how our training objective supports minimal sufficient editing.

Our notion of minimality is instruction-relative: complete the requested edit while preserving content that need not change. We pursue this implicitly through paired-target training (main paper Section 3.4). The full-image flow-matching loss supervises the entire edited target, including outside the compiled support. Wherever that target preserves source content, the loss also supervises preservation. SCSL strengthens target supervision within the support to discourage source copying and incomplete edits, while the global term remains active. Together, these terms encourage both edit sufficiency and preservation.

An explicit minimality constraint that generalizes across our editing regimes would need to identify which semantic changes are necessary for each instruction. A fixed pixel-distance or outside-mask identity penalty cannot capture this consistently: global stylization can alter the whole image while retaining identity and composition, and an object replacement may require changing its silhouette. Our paired targets demonstrate desired edits but do not rank all valid outputs by pixel-level minimality.

AuthorsOpenReview ↗

Reply to Reviewer FkRb (part 3)

5. [Q6, W2]: Which modules were retrained for Table 3 of the main paper, and why does removing focal-support expansion or an SES field hurt performance?

A: We thank the reviewer for highlighting the need to explain these ablations and document their settings more clearly.

Focal-support expansion. Without expansion, the tight source-object mask can leave insufficient room for the boundary adjustments needed to integrate the edited object with its background. Even when the requested semantic change is recognizable, restricting synthesis support to the original silhouette can produce inconsistent object–background transitions and impair perceptual preservation. The expanded support provides a surrounding margin for these necessary adjustments. This is consistent with Table 3(b) of the main paper: ER-IF remains close to the full model (1.589 vs. 1.630), while InvPres drops from 0.896 to 0.205 and BG-LPIPS increases from 0.150 to 0.337.

SES-field ablations. The fields have distinct computational roles: Planning selects the editing regime, Action provides semantic guidance, and Layout determines spatial conditions. Perceiving describes the source-to-target states before Action and Layout are generated autoregressively (main paper Section 3.2). Omitting this step during generation can therefore affect subsequent semantic and spatial planning, even though Perceiving is not a separate executor input. Its influence is mediated through the later fields used to construct the intervention conditions.

Retraining protocol. We thank the reviewer for requesting this clarification. Both the planner and the editor were retrained for every ablation variant in Table 3 of the main paper under the corresponding ablated configuration. Apart from the component under study, the remaining architecture and loss definitions were kept unchanged. For the “w/o SCSL” variant, only SCSL was removed from the training objective. We will explicitly state this retraining protocol in the revised manuscript and the caption of Table 3 of the main paper.

6. [Q5, W2]: How do predicted plans compare with oracle plans, and what is the exact edit-type accuracy?

A: We thank the reviewer for suggesting this informative diagnostic.

The results in the main paper use predicted SES at inference. We additionally evaluated ground-truth SES on the same test samples, keeping the trained editor, compilation pipeline, and generation settings fixed. Comparing these two conditions using the same editing metrics quantifies how errors in the predicted plans affect downstream instruction fulfillment and preservation. The oracle setting serves as a diagnostic of the editor’s performance when provided with annotated plans.

Test-time plan (same editor)Edit-type Acc. (%) ↑ER-IF ↑ETP ↑InvPres ↑BG-LPIPS ↓Int-F1 ↑ER-VQ ↑
Predicted SES (MSVI, reported in Table 1)83.41.6300.4640.8960.1500.4640.012
Ground-truth SES (oracle)N/A (GT supplied)1.7620.4770.9040.1470.4720.014

Table A-3. Comparison of predicted and ground-truth SES on the same test inputs, with the trained editor, compilation pipeline, and generation settings held fixed. Edit-type accuracy measures exact agreement with the annotated editing regime.

As shown in Table A-3, predicted SES achieves 83.6% edit-type accuracy and approaches oracle performance in both instruction fulfillment (ER-IF: 1.630 vs. 1.762) and preservation (InvPres: 0.896 vs. 0.904), supporting the practical effectiveness of the learned planner despite the use of predicted plans at inference.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reply to All Reviewers

We are grateful for the positive feedback and constructive suggestions from the reviewers. For each item, we have provided detailed responses and supplemented necessary experiments. In main paper, we have also added necessary sections for experiments, such as 4.4 End-to-End Inference Latency and 4.5 Human Evaluation. If you have any further questions, please feel free to let us know. We are more than willing to have more discussions. Thanks.

-Authors of paper 17382

AuthorsOpenReview ↗

Reply to All Reviewers

Dear Reviewers,

We have provided our responses to all of your questions and comments on the paper. As the rebuttal period is drawing to a close, please let us know if there is anything further you would like us to clarify, we would be happy to provide any additional details.

Sincerely,

-The authors of paper 17382

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.