acceptodds
Under review as a conference paper at ICLR 2027

ControlEdit: Image Editing with Unpaired Post-Training

Abstract

Recent instruction-based editing models achieve impressive editing performance by fine-tuning pretrained text-to-image models with large-scale pair (source image, target image) editing data. However, constructing such pair editing data is expensive, often requiring a complex data pipeline and human verification, which makes the process of finetuning text-to-image model to image editing model costly and difficult. In this work, we propose \method, a post-training framework that converts a pretrained text-to-image model into an instruction-based image editing model without requiring target image during training. Our framework consists of two stages. The first stage converts text-to-image diffusion model to take instruction and source image as editing condition, while the second stage applies reinforcement learning post-training to further optimize editing quality with respect to prompt alignment and background preservation. Experiments across multiple editing tasks demonstrate that \method achieves competitive performance with supervised editing methods while eliminating the need for paired edited images.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Image Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MpaJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • ControlEdit adapts a text-to-image model without direct supervision from edited target images. Reconstruction-trained ControlNet features provide source conditioning, followed by instruction distillation and reinforcement learning with adaptive conditioning strength.
Critical strengths
  • The method offers a practical way to reduce dependence on target editing images.
  • Comparisons with supervised models using the same architecture are useful, and component ablations support combining adaptive conditioning with LoRA updates.
Critical weaknesses
  • Baseline fairness is not fully established: most models use ten sampling steps, while the supervised counterpart's training budget is insufficiently specified.
  • Avoiding target-image supervision does not by itself establish lower overall cost. The method uses an existing paired dataset and substantial RL sampling.
Questions for authors
  1. Clarification: How are the target descriptions used during instruction distillation obtained? Does their construction require access to edited target images?
  2. Evidence request: What are the supervised baseline's training budget and tuning procedure? Do the main comparisons hold when competing models use their recommended sampling configurations?
2 comments

We thank reviewer MpaJ for valuable feedback. We address the reviewer's concerns below:

  1. Weakness 1 and Question 2: Supervised training budget and other methods' default configuration

Supervised training budget: Our supervised model uses the same configuration as ControlEdit, including the same ControlNet architecture and a LoRA rank of 128, but without the gating network. The supervised model is trained on the same GPT Image 1.5 M dataset, using ground-truth edited images as supervision. We train the supervised model with a batch size of 8 per GPU on 4 A100 GPUs for 40k iterations, requiring 118 GPU hours.

Sampling configuration of baseline models: The Step1X-Edit and Qwen-Image-Edit baselines use their respective default numbers of sampling steps, 28 and 50. The full results are presented below:

MethodBase ModelTraining#Params#StepsSC ↑PQ ↑Overall ↑
Original sampling steps
Step1X-EditFLUXSupervised12B287.617.347.24
Qwen-Image-EditQwen-ImageSupervised20B507.207.566.97
Reduced sampling steps
Step1X-EditFLUXSupervised12B107.617.127.16
Qwen-Image-EditQwen-ImageSupervised20B106.977.496.97
FLUX.1 KontextFLUXSupervised12B106.837.156.49
NP EditZ-ImageUnsupervised6B45.657.365.50
ControlEdit-M (ours)SD3-MediumUnsupervised2B106.127.176.01
ControlEdit-L (ours)SD3-LargeUnsupervised8B106.327.386.23

As the number of sampling steps increases, both Step1X-Edit and Qwen-Image-Edit show consistent improvements in editing performance. Our method underperforms these methods, both with the same number of sampling steps and under their default configurations. In our paper, we only claim that ControlEdit outperforms NP Edit in terms of SC and PQ scores.

We will add the refined Table to our revision of this paper.

  1. Weakness 2 and question 1: Target description and RL cost concerns:

Target description concerns: ControlEdit does not use target images at any stage. It only requires the source image, editing instruction, and source/target prompts, none of which require access to the edited image. The editing instructions and target prompts can be generated by a language model from the source prompt or source image alone, as demonstrated by InstructPix2Pix, which generates both the editing instruction and edited caption from the input caption before any image synthesis, and NP Edit, which uses a VLM to generate editing instructions and edited-image captions directly from reference images. Therefore, our training data can be constructed from any image corpus without requiring the generation, filtering, or verification of edited images.

Cost for ControlEdit and comparison with the cost of data pipeline: We agree with the reviewer that RL post-training is a computationally expensive stage. Our DiffusionNFT training requires 1,038 GPU hours, while ControlNet pretraining and instruction distillation require 7 and 12 GPU hours, respectively. However, we argue that this cost is still lower than that of an edited-image generation pipeline. To illustrate this, we consider a simple strategy using Qwen-Image-Edit with 50 sampling steps as the data generator. Generating a 512×512 image with Qwen-Image-Edit takes 28 seconds on 2 A100 40GB GPUs, corresponding to approximately 56 GPU-seconds per image. Our training consists of 30k iterations, covering approximately 480k images. Generating the same number of edited images using Qwen-Image-Edit would require more than 7,000 GPU hours, which is substantially more expensive, even without accounting for the verification stage.

Solved my concerns, so I will keep my rating.

Reviewer_bDPFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ControlEdit adapts a pretrained text-to-image flow model into an instruction-based image editor without using paired target edited images as training supervision. At inference, it takes a source image and an editing instruction and produces an edited image intended to satisfy the instruction while preserving unrelated content. Training combines source-image reconstruction through ControlNet, instruction distillation from target-prompt predictions, and VLM-reward post-training of LoRA parameters and layer-specific conditioning gates. The paper evaluates SD3-Medium and SD3-Large variants on GEdit-Bench and ImgEdit.

Critical strengths

The data requirement is worth investigating. Learning an editor without paired target images addresses a real obstacle in constructing editing supervision.

The reported experiments provide useful empirical observations. Both model sizes improve over their supervised architectural counterparts under the reported GEdit setting, and the second benchmark and component ablations help characterize the resulting behavior.

The limitations are partially documented. Task-level results and the acknowledged object-removal failures help delineate what the method can and cannot currently achieve.

Critical weaknesses
  1. The practical value of the additional ControlNet is insufficiently established. Encoding source images through a separate conditioning branch is a mature design paradigm. Against existing unified generation-and-editing architectures, the paper does not establish sufficient practical value for separately training a ControlNet.

  2. The NFT stage provides limited methodological novelty. DiffusionNFT is an existing optimization method, and applying it to this pipeline does not by itself constitute a new contribution. The paper does not establish a substantive modification or a new training principle beyond combining established components.

  3. Figure 2 is poorly presented. Its coarse imagery, decorative styling, and underspecified connections make the training stages and information flow difficult to follow. The central method figure needs a clearer, higher-quality rendering with consistent notation and explicit conditioning and parameter-update paths.

Questions for authors

What concrete benefit does a separately trained ControlNet provide over a unified image-conditioned backbone for this setting? Can that benefit be demonstrated while controlling the backbone, available supervision, training budget, and inference cost?

What is technically new in the NFT stage beyond applying DiffusionNFT? Please isolate any proposed modification from the standard algorithm, and clarify the inconsistent negative-policy expressions in the text and pseudocode.

Can Figure 2 be replaced with a precise, high-quality diagram covering the reconstruction, instruction-distillation, and post-training stages, including timestep conditioning, the reward model's inputs, and all trainable-parameter update paths?

4 comments

We thank reviewer bDPF for the valuable feedback. We address the reviewer's concerns below.

  1. Weakness 1 & Question 1: Comparison between the choice of ControlNet and the unified token concatenation conditioning mechanism:

Choice of ControlNet instead of token concatenation:We agree that token concatenation is a strong and widely adopted design in image editing. However, our design specifically targets the unpaired setting, where ControlNet trained with reconstruction supervision can serve as an initial editor.

Benefit of ControlNet in the unpaired setting: With a separate ControlNet branch, source preservation is handled by ControlNet, while the frozen T2I prior remains capable of following a new prompt. As a result, editing capability emerges directly from reconstruction training (Fig. 1), without requiring additional regularization. The ControlNet branch also enables control over the preservation-editing trade-off through the conditioning scale ss, which our gating network learns to adjust for each instruction and layer. In contrast, when applying the same reconstruction training to concatenation-based conditioning, we observe that the model tends to reproduce the source image and requires additional regularization (source-conditioning dropout and a directional loss, Appendix A.2) to retain its editing capability.

Comparison with the concatenation regarding the training budget and inference time: Appendix A.2 (Table 12 and Figure 5) compares both settings with the same training budget. Under our current training strategy, the ControlNet design achieves 6.01 Overall and 7.17 PQ, compared with 5.71 and 6.48 for concatenation, which shows more hallucinated content (Fig. 5). We attribute this to how the two designs allocate responsibilities. In the ControlNet design, preservation is handled by the ControlNet branch, while the LoRA adapts the transformer for editing. In the concatenation design, the same LoRA parameters must handle both preserving the source and applying the edit, which, without paired targets to separate the two, makes it more prone to altering unrelated regions or introducing spurious content. Below is the inference time comparison on one 512x512 image on a single A100 40GB GPU:

ConditioningSC ↑PQ ↑Overall ↑Latency per image ↓Peak memory ↓
ControlNet + gating (ours)6.127.176.010.93 s17.37 GiB
Token concatenation5.946.485.711.02 s13.63 GiB

The table shows that the two designs have comparable computational costs, with different trade-offs: the ControlNet branch introduces additional parameters, increasing peak memory usage, while token concatenation doubles the token sequence length processed by self-attention, resulting in higher inference latency.

We will move Appendix A.2 to the main paper for better clarification in our revision.

  1. Weakness 2 and Question 2: Concern about limited novelty of DiffusionNFT:

Concern about Diffusion NFT: We agree that DiffusionNFT is an existing framework. However, we would like to clarify that we do not claim DiffusionNFT as a contribution of our paper. Our main contribution is the observation that editing capability emerges from ControlNet reconstruction training, which allows us to decompose unpaired image editing into two subtasks: prompt alignment and background preservation. The RL stage is introduced to further improve this emergent editing capability. We chose DiffusionNFT because it is a standard RL post-training algorithm that can be applied to a pretrained model (in our case, the Stage 1 model) to improve its reward.

Compared to standard DiffusionNFT, our training jointly optimizes the gating network and the LoRA parameters of the diffusion transformer. Additionally, we adopt the VLM score from NP Edit as the reward function.

Inconsistency in the negative policy expression: We thank the reviewer for catching this error. Eq. 7 is correct: v−=(1+β)vθold−βvθv^- = (1+\beta)v_{\theta_{\text{old}}} - \beta v_\theta, which is consistent with both DiffusionNFT and our implementation. Line 33 of Algorithm 1 contains a typo and should instead read v−=(1+β)vold−βvfwdv^- = (1+\beta)v_{\text{old}} - \beta v_{\text{fwd}}. We will correct this in the revised manuscript.

  1. Weakness 3 and question 3: Figure 2 weaknesses

We thank the reviewer for valuable feedback. We will revise Figure 2 and provide a more comprehensive visualization in our revision.

Thank the authors for the clarification and additional efficiency measurements. My main concern is the contribution and practical value of the overall approach.

Regarding Q1, the token-concatenation ablation supports choosing ControlNet within your current reconstruction and post-training recipe. However, my question is what important limitation of existing image-conditioned editing approaches does the additional ControlNet branch resolve? Source-image conditioning is already a mature capability, so its value needs to be established through a meaningful advantage in the unpaired setting—for example, better editing quality under comparable supervision and resources, or substantially lower training cost at comparable quality. The current comparison shows that your ControlNet variant works better than your concatenation variant, but does not sufficiently establish such an advantage over the closest existing unpaired editing approaches.

I understand that DiffusionNFT itself is not claimed as a contribution. The question is what substantive novelty the reconstruction-based initialization and adaptive gating provide beyond combining established components. The response does not yet sufficiently establish this novelty or its practical significance.

Moreover, manuscript revisions are permitted during the current discussion period, but I have not seen corresponding updates addressing the reviewers’ concerns, including the promised correction to the algorithm and revision of Figure 2.

I therefore maintain my overall assessment.

Thank the reviewer for the follow-up; we provide clarification to address the reviewer's concern.

Q1. We want to clarify that ControlNet is not our contribution. It is a module that enables the decoupling of the editing objective in our framework: editing alignment and background preservation.

Compared with NP Edit: Compared with NP Edit. Under comparable supervision (no target images, VLM-based feedback), ControlEdit-M outperforms NP Edit’s released checkpoint in SC (6.12 vs. 5.65) and Overall (6.01 vs. 5.50) on GEdit-Bench. We found that NP Edit often fails to execute the instruction and sometimes returns an image nearly identical to the source. On GEdit-Bench, NP Edit receives an SC score of 0 on 159 samples (26.2%), compared with 107 (17.7%) for ControlEdit, and 41 of its outputs (6.8%) are nearly identical to the source image (LPIPS < 0.05), compared with only 1 (0.2%) for ControlEdit. We attribute this to how the edit-preservation trade-off is controlled. In NP Edit, the trade-off is implicit in a single fine-tuned transformer. In ControlEdit, the decomposition exposes it explicitly through the ControlNet scale, which the gating network adapts per instruction and per layer, allowing stronger edits where needed while preserving the rest.

Q2. Reconstruction-based initialization: RL post-training typically starts from an SFT policy. For editing, however, SFT requires target images, which are unavailable in the unpaired setting. We find that a ControlNet trained only for source-image reconstruction already exhibits an emergent editing capability, which can replace the SFT model as the initialization for RL. As shown in Table 3, this reconstruction-based model reaches an Overall score of 5.42 on GEdit-Bench before RL post-training.

Adaptive gating network: In existing ControlNet usage, the conditioning scale is a fixed inference hyperparameter set by hand. We make it part of the RL policy: the gating network predicts the scale per instruction, per layer, and per timestep, and is optimized from a non-differentiable reward. This removes manual tuning across edit types and outperforms the best fixed scale (6.01 vs. 5.86, Table 2), improving editing strength while maintaining preservation.

Q3. We have updated the paper, including Figure 2.

The rebuttal clarifies the choice of ControlNet within the proposed training recipe and provides useful inference measurements. However, it does not sufficiently establish the approach’s practical advantage over existing unpaired editing methods or its substantive novelty beyond combining established components. I therefore maintain my original score.

Reviewer_EPxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ControlEdit, an unpaired post-training framework for instruction-based image editing. The main motivation is to reduce the dependence on expensive paired source-target editing images. The method uses a pretrained text-to-image model for prompt alignment and ControlNet for preserving the source image. It first trains ControlNet with source-image reconstruction and performs instruction distillation, then introduces an adaptive gating network and RL post-training with VLM-based rewards. Experiments on SD3-Medium and SD3-Large show competitive editing performance without directly using paired target images for training.

Critical strengths

Collecting high-quality paired editing data is expensive. The paper explores an alternative training strategy without target-image supervision, which is a meaningful direction for image editing.

ControlEdit achieves better results than its supervised counterpart under the same architecture and outperforms NP Edit on the overall GEdit-Bench score. This shows the potential of VLM-reward-based RL post-training for image editing.

Critical weaknesses

the method mainly combines existing components, including ControlNet, instruction distillation, adaptive conditioning, and DiffusionNFT. Although the overall framework is reasonable, the technical novelty is relatively limited, and the paper does not sufficiently explain why this particular combination is better than other possible unpaired training approaches.

ControlEdit performs poorly on object removal compared with existing supervised and unsupervised methods. For example, its object removal SC score is only 3.4 on GEdit-Bench, compared with 7.2 for Step1x-Edit and NP Edit. This suggests that the method still has difficulty handling some important editing operations.

Limited comparison with frontier image editing models: Although the paper compares with several strong open-source methods, it does not include frontier image editing models such as GPT-Image2 or nano-banana image editing systems.

Questions for authors

na

2 comments

We thank Reviewer EPxs for their valuable feedback. We address the reviewer's comments and concerns below.

Weakness 1: Concern about novelty and the choice of design: "The method mainly combines existing components, including ControlNet, instruction distillation, adaptive conditioning, and DiffusionNFT. Although the overall framework is reasonable, the technical novelty is relatively limited, and the paper does not sufficiently explain why this particular combination is better than other possible unpaired training approaches".

1.1. Novelty and explain our choice of design: We agree that the individual components have been explored in previous work, and we do not claim them as our main contributions. Instead, we would like to clarify our main contribution: the emergent editing capability of a ControlNet conditioning model trained solely on source image reconstruction, which can perform image editing when provided with a target prompt. This finding allows us to decompose image editing into two subtasks: prompt alignment and background preservation, with each component serving a specific purpose:

  • ControlNet: Responsible for background preservation.
  • Instruction distillation: Transfers the editing capability from target-prompt-based editing to instruction-based editing.
  • Adaptive conditioning: Because preservation is isolated in a ControlNet path, the alignment–preservation trade-off reduces to the conditioning scale s, whose optimal value depends on different image-instruction pairs. This motivates the adaptive gating network rather than a fixed global scale.
  • DiffusionNFT: Performs RL post-training to further improve the editing capability of the distilled model, using a VLM scoring judgment.

1.2. Compared with other unpaired edit approaches:

Conditioning mechanism: Both NP Edit and Bootstrap Generator adopt token concatenation for image conditioning. We also evaluate token concatenation as an alternative to ControlNet in ControlEdit (Table 12). Under an identical unpaired training pipeline, replacing ControlNet with token concatenation reduces Overall from 6.01 to 5.71 and PQ from 7.17 to 6.48 (Table 12), with visibly more hallucinated content (Fig. 5). We attribute this to the entanglement of editing and preservation: with token concatenation, a single adapted transformer must both preserve the source image and follow the editing instruction, whereas our design assigns these roles to separate modules. Notably, the concatenation variant requires additional regularization (source-image conditioning dropout and a directional loss, Appendix A.2) to avoid simply copying the source image during reconstruction, yet still underperforms. In contrast, our ControlNet-based decomposition requires no such measures. Developing training objectives that make concatenation-based conditioning effective in the unpaired setting is an interesting direction that we leave for future work.

Training cost: NP Edit fully fine-tunes the diffusion transformer, trains an additional full-size auxiliary network for its DMD objective, and backpropagates the VLM score through the VLM. Its authors also acknowledge the resulting VRAM overhead as a limitation. In contrast, ControlEdit trains only a lightweight ControlNet, a gating network, and LoRA parameters, while using the VLM solely as a forward-pass reward within a standard RL framework. To quantify the memory efficiency, we measure peak allocated GPU memory per training step (maximum across 4 GPUs) on RTX A6000 GPUs under matched settings (SD3-M backbone, Qwen3-VL-4B, resolution, precision, and gradient checkpointing). With a per-GPU batch size of one source-instruction pair, the NP Edit-style step requires 38.1 GB, even when using LoRA instead of full fine-tuning, which underestimates its actual memory footprint. In comparison, ControlEdit requires only 30.4 GB, approximately 20% less memory. Furthermore, with a 2B backbone, ControlEdit-M outperforms the released NP Edit checkpoint (6B) in both SC and Overall (Table 1).

We will move Appendix A.2 to the main paper for better clarification in our revision.

Weaknesses 2: Weaknesses for removal task: We agree that object removal is ControlEdit's main weakness. In our framework, editing emerges from a target prompt that specifies content the T2I model can generate. However, for removal tasks, the target prompt simply omits the object, making it difficult for the T2I model to understand the intended removal. RL also struggles to address this limitation, as effective RL training requires reward variance, while nearly all rollouts fail to remove the target object. Nevertheless, ControlEdit outperforms NP Edit on other categories, including background, material, motion, human editing, and style (Table 5). Improving removal performance by increasing the success rate of removal rollouts is a natural direction for future work, which we will discuss in the revised paper.

Weakness 3: Frontier comparison

We thank the reviewer for the feedback. We additionally include a comparison with Nano Banana 2.1 in the table below:

MethodBase ModelTraining#ParamsSC ↑PQ ↑Overall ↑
Nano BananaProprietaryProprietaryUndisclosed7.857.577.55
Step1X-EditFLUXSupervised12B7.617.127.16
Qwen-Image-EditQwen-ImageSupervised20B6.977.496.97
FLUX.1 KontextFLUXSupervised12B6.837.156.49
NP EditZ-ImageUnsupervised6B5.657.365.50
ControlEdit-M (ours)SD3-MediumUnsupervised2B6.127.176.01
ControlEdit-L (ours)SD3-LargeUnsupervised8B6.327.386.23

We also include this result in Table 1 of the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.