acceptodds
Under review as a conference paper at ICLR 2027

Can We Perform Online RL for Image Editing without Editing Rewards?

Abstract

Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly `edit instruction, source & edited image' triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph rendering, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences for RL-based optimization, raising the natural question: Can We Perform Image Editing RL without Editing Rewards? In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, building such lever is non-trivial. Editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically correct captions to humans may not induce the frozen reward to rank successful edits above failures. We therefore introduce Lever-Edit, a two-stage reward-transfer framework. Stage 1 learns a textual reward interface that produces both semantically faithful and reward-aligned descriptions of desired post-edit states with a pluggable target-state adapter, change-aware source–target contrast, two-level preference calibration, and regularizations. Stage 2 freezes this interface and optimizes editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines {across various benchmarks.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PygnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether image editing RL can be performed without a dedicated editing reward. It trains a Lever module that converts a source image and an editing instruction into a complete target-state description, and then uses mature T2I reward models to evaluate edited images. Lever training combines target-state alignment, source-target contrast, preference calibration, and semantic/distributional anchoring.

Critical strengths

The paper studies an interesting and potentially impactful problem: whether the mature T2I reward ecosystem can be reused for image editing, thereby reducing the cost of constructing dedicated editing rewards. The Lever-Calibration and Lever-Anchoring components provide a useful discussion of reward alignment and reward hacking. Experiments with multiple editing backbones and captioners provide some evidence of transferability.

Critical weaknesses
  1. A Lever-generated caption may still describe the global image correctly, while the T2I reward only evaluates overall image-text alignment. It may not understand that a specific local region should remain unchanged. The paper should construct fine-grained source-edit consistency tests and compare the transferred T2I reward with a genuine source-edit reward.

  2. The claim that editing rewards are unnecessary is somewhat broader than what the experiments establish. Lever training still relies on editing data, successful/failed edit pairs, and editing-specific calibration examples. The evaluation also uses editing-specific metrics such as EditScore. A comparison with a directly trained editing reward under matched data and computational budgets would clarify the actual savings.

Questions for authors

See Critical Weaknesses

1 comment
AuthorsOpenReview ↗

Dear Reviewer Pygn,

We sincerely appreciate your time and effort in reviewing our work. We have done our best to address the concerns you raised, and we would greatly appreciate it if you could kindly share your thoughts when time permits.

Reviewer_9KZgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Lever-Edit, a framework for adapting existing text-to-image reward models to online reinforcement learning for image editing. Its central idea is to translate a relative editing instruction, together with the source image, into an absolute description of the intended edited image. A vision-language model is first calibrated to produce descriptions that work well with frozen reward models, using alignment, edit-aware preference objectives, and semantic and distributional anchors. The resulting descriptions are then frozen and cached while the editor is optimized using existing text-to-image rewards.

The paper evaluates the approach on the proposed Lever-Bench and several public editing benchmarks, primarily with FLUX.1-Kontext-dev, and presents extensions to additional editors and description models. The reported results support improved instruction execution and demonstrate that reward-oriented calibration of the description interface can outperform straightforward instruction rewriting.

Critical strengths
  1. The paper identifies a practically important mismatch between editing instructions and existing reward interfaces. A relative instruction often leaves most of the desired image implicit, which makes its direct use with a text-to-image reward problematic. Learning an absolute target description is a useful way to address this mismatch. The distinction between a fluent description and a reward-compatible description is also meaningful, and the semantic and KL anchors provide a concrete response to reward-driven distortion of the text interface. Freezing and caching descriptions before editor training makes the approach operationally attractive.

  2. The instruction-execution gains are supported by an evaluator that is not the online training reward. In Table 1, EditScore execution improves from 0.6311 for the base editor and 0.7191 for Offline-SOTA rewriting to 0.8027 for Lever-Edit. Table 3 also shows a substantial improvement when edit-aware calibration is added to anchored absolute alignment. These results provide substantive evidence beyond increases in HPSv3 or RealGen, which are directly optimized. The additional public-benchmark evaluations strengthen the case that the approach addresses a useful editing problem.

Critical weaknesses
  1. The provenance and separation of calibration, editor-training, and evaluation data are insufficiently specified. Lever-Bench is introduced as an evaluation benchmark in Section 4.1, while Appendix B.1 states that preferred–failed calibration pairs originate from the Lever-Bench building process. The appendix also describes 500 calibration inputs, but does not provide a complete overlap accounting among these inputs, preference-pair construction, the semantic-anchor data, the Stage 2 editing dataset, checkpoint selection, and evaluation. The Stage 2 dataset itself is not concretely identified. In addition, the stronger aligned editor used to obtain preferred outputs and the amount of supervision it contributes are unclear. This does not establish evaluation leakage, but it prevents a reliable interpretation of the large Lever-Bench gains as held-out performance and leaves the total editing-specific supervision required by the method uncertain. This is my main obstacle to acceptance.

  2. The evidence for source-image preservation is less conclusive than the evidence for instruction execution. The SigLIP measure described as consistency evaluates image–text alignment, rather than preservation of source-image content. Table 2 usefully includes fixed-caption comparisons, but also shows that caption choice alone changes the score: for the same base output, replacing Pure-Edit text with Lever-Edit text changes the score from 0.0769 to 0.0784. Holding the Lever-Edit text fixed, changing the output from the base editor to Lever-Edit increases it from 0.0784 to 0.0788. These are valid alignment results, but they should not be interpreted as direct evidence of detailed source preservation. The independent EditScore consistency result is encouraging relative to the base editor, although Lever-Edit does not exceed Offline-SOTA on consistency or quality in Table 1. The small overall margin in Figure 5(b), 5.35 versus 5.32, is also reported without uncertainty. Consequently, the broader quality and preservation claims are less securely established than the execution improvement.

Questions for authors
  1. [Clarification] Please provide a data-flow and split table identifying the source, sample count, and overlap status of the Stage 1 calibration inputs, preferred–failed pairs, semantic-anchor examples, Stage 2 editing inputs, checkpoint-selection data, and every evaluation set. In particular, do any Lever-Bench evaluation inputs participate in calibration or preference-pair construction?

  2. [Clarification] Which stronger aligned editor produces the preferred outputs, how was it aligned, and how many preferred–failed pairs are used? Please also specify the size and construction cost of the semantic-anchor dataset. Did either upstream component use editing-specific rewards or preference supervision?

  3. [Assessment-relevant validation] On a clearly disjoint evaluation set, how robust are the improvements over offline rewriting and the strongest relevant editing baseline under a shared evaluation protocol with paired uncertainty? A source-sensitive preservation assessment, alongside instruction execution, would materially affect my judgment. The existing fixed-caption analysis in Table 2 is useful but does not resolve this question.

Reviewer_7JZUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Proposes Lever-Edit, a two-stage framework transferring mature T2I rewards to image-editing RL without editing-specific rewards. Stage 1 trains a VLM captioner ("Lever") via GRPO to map (I_ref,q_edit) to a reward-aligned absolute target caption, using a composite calibration reward (absolute alignment, change-aware contrast, no-op and preferred-vs-failed preferences) plus semantic and KL anchors. Stage 2 freezes the Lever and optimizes an editing diffusion backbone with only frozen T2I rewards (HPSv3, RealGen). Contributes a failure-focused Lever-Bench (1,880 pairs) and shows gains over direct-transfer baselines and competitiveness with EditScore.

Critical strengths
  1. Crisp problem framing: the relative-instruction vs. absolute-condition mismatch is well articulated, and the quality→T2I-space mapping is clean.

  2. Practical decoupling: frozen Lever in Stage 2 allows precomputed captions, no repeated VLM inference; pluggable across backbones (FLUX-Kontext, BAGEL, Qwen-Image-Edit) and captioners.

  3. Convincing reward-hacking diagnosis: Table 5's "addCriterion / muted-toned" degeneration without anchoring is the paper's strongest empirical evidence; Table 3 cleanly separates anchoring vs. edit-aware reward effects.

  4. Multi-level evaluation: separates description quality from image quality (Table 2), and shows offline GPT captions are not necessarily reward-compatible — a useful lesson.

Critical weaknesses
  1. Overclaimed title: "without editing rewards" hides real costs — a learned Lever, a GPT-5.5 teacher, curated preference pairs, and four hand-tuned reward terms. Cost is moved, not removed; no accounting vs. training an EditScore-style reward.

  2. Missing baselines: no trivial "caption reference + append instruction" baseline to isolate Stage 1's value; no distillation baseline (T2I reward → editing reward).

  3. Consistency claims inconsistent: Sec. 4.2 says Lever-Pure-Edit has a "pronounced drop in consistency," but Table 1 shows its SigLIP consistency (0.0772) > w/o Alignment (0.0769) and near Lever-Edit (0.0788).

  4. Narrow reward coverage: OCR/safety/zero-shot rewards only qualitative (Fig. 10); no failure taxonomy of non-transferable T2I rewards.

Questions for authors
  1. Quantify total Lever-Edit cost (Stage 1 compute, teacher calls, pair curation) vs. training an editing reward.

  2. How does it compare to a simple absolute-conditioning baseline (caption reference + append instruction) fed to the same T2I rewards?

  3. Reconcile the Lever-Pure-Edit consistency claim with Table 1's SigLIP numbers.

  4. Report sensitivity to the RealGen/HPSv3 mix and the four λ weights; does the optimum vary by backbone/edit type?

  5. Give a quantitative negative case: a T2I reward whose preferences are misaligned with editing.

  6. What is human agreement with GPT-5.5 failure labels on Lever-Bench? Do gains hold on uniformly sampled edits?

  7. Report mean ± std over ≥3 seeds for Table 1 and Fig. 5.

1 comment

Dear Reviewer 7JZU,

We sincerely appreciate your valuable feedback. We understand that you may have a busy schedule and would be grateful if you could discuss our work with us, should you have the opportunity. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.