acceptodds
Under review as a conference paper at ICLR 2027

VibeEdit: Image Editing with Canvas Instructions

13 views

Abstract

In text-guided image editing, describing the desired change is often straightforward, but identifying the intended object or region can be cumbersome, especially when several objects look alike. We introduce a new image editing interface that lets users place spatial marks and optional short notes directly on the image. Together, these annotations form a *canvas instruction* that specifies where to edit and what to change. Our editor, *VibeEdit*, follows these instructions to perform object addition, removal, replacement, attribute modification, and movement without a separate text prompt. We construct 1.55 million source-target edit pairs with object masks and structured edit descriptions, from which we render canvas instructions during training. We adapt Qwen-Image-Edit with layer-decoupled conditioning that separately encodes source images and canvas instructions for image editing. We train the model with region-weighted supervised fine-tuning, followed by rubric-guided reinforcement learning to improve edit completion, local edit quality, and preservation of unedited regions. We evaluate VibeEdit on an independently constructed, human-curated benchmark of 419 cases emphasizing target selection among similar objects. VibeEdit achieves a VLM rubric score of 79.9 and an outside-region PSNR of 32.8 dB, compared with 67.4 and 24.0 dB for FireRed, the highest-scoring text-instructed baseline in our evaluation.

Primary area:
generative models
Keywords:
Image Edit, Diffusion, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mQq1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an editing pipeline which takes user annotations on the image as condition. The annotations form a canvas instruction that specifies where to edit and what to change. A dataset is constructed in order to enable the training, which covers the common editing types like adding, removal, replacement and so on. Qwen-Image-Edit is adapted to take the canvas as condition, and rubrics-based RL is further applied after the finetuning to improve the editing quality. Results show that the proposed pipeline can take the instructions on canvas as input and correctly perform the edits than general baselines.

Critical strengths
  1. The pipeline is very clear, and every step is cleanly explained and motivated. The results also show that the model can recognize the instructions on the canvas and perform the edits inside the indicated region.
  2. The data construction pipeline is reasonable. It also covers the most common types of edits.
Critical weaknesses
  1. It seems that substantial quality for the evaluation is judged by VLM rubrics score, which appears to be the same reward as what's used in the proposed RL finetuning stage. This might give some fairness concerns for the other baselines which did not finetune using the rubrics score.
  2. It's less surprising that feeding the canvas-annotated images as input to general backbones like Qwen will cause instruction-retention, since those models are not finetuned with these specific canvas images. I understand that the quantitative numbers are both provided for visual-only setup and text-instructed setup. However, for the visual comparison, what's more interesting and more fair is to show the results where for general backbones, the source image + text instruction are used as input. This can show a fair performance comparison.
  3. One of the potential use case for on-canvas instruction is multi-edit, where there are multiple areas the users intend to edit, and it's easier for the users to tell the model which instruction is for which area. However, multi-edit results are not shown. Otherwise, I currently could not think of the advantages of having instruction on the image. Since this is part of the main contribution of the paper, I'm open to discussions about this with the authors.
Questions for authors
  1. Does the RL training stage share the same dataset / training examples as the previous finetuning stage?
  2. As GPT-5.6-sol is used throughout most of the steps, why is instead GPT-5.4-mini used in the RL stage?
  3. In dataset construction stage, for object replacement, how is the expanded version of M generated? Does the model have difficulty to handle replacement large size mismatch during inference? Dose the model have difficulty to preserve the background when the target object is much smaller than the original size?
5 comments

W1: Fairness of rubric-based evaluation after RL training

Thank you for this thoughtful point. First, we would like to highlight that even before RL, VibeEdit-base already outperforms all baselines in Table 1. Second, because SFT relies on a proxy objective (flow-matching MSE) that does not directly measure instruction following or visual fidelity, RL post-training is intended to align the model with what human users actually care about at test time. Using RL rewards that mirror target evaluation criteria to align with human preference is standard practice in both LLM and diffusion RL literature. For example, in LLMs, DeepSeek-R1[1] optimizes rule-based accuracy rewards evaluated on math and coding benchmarks, and Rubrics as Rewards[2] optimizes LLM-judged checklists evaluated on rubric benchmarks like HealthBench; in visual generation, Flow-GRPO[3], DanceGRPO[4], and DiffusionNFT[5] directly optimize target evaluation metrics such as GenEval, OCR, HPSv2.1, and PickScore. Following this practice, we design both our RL reward and benchmark rubric around edit completion, background preservation, and local quality, pairing a lighter RL judge (GPT-5.4-mini) with a stronger test judge (GPT-5.6-sol) alongside outside-region PSNR (which rises from 27.0 to 32.8 dB).

Besides, to give the baselines a completely fair, rubric-free comparison, we conducted a blind user study with 10 human raters over 500 pairwise comparisons, evaluating VibeEdit and all six open-source baselines from Table 1 in their stronger text-instructed setting. Human raters were not shown our rubric questions and judged instruction following and visual quality directly. We report strict win rate, Elo score (ties counted as 0.5), and the win/tie/loss match counts:

ModelWin RateElo
Qwen-Image-Edit-251147.1%1543.8
LongCat-Image-Edit42.7%1525.8
FLUX.2-klein-9B37.6%1420.0
FireRed-Image-Edit-1.041.7%1506.5
JoyAI-Image-Edit41.7%1413.6
Step1X-Edit-v1p243.8%1435.5
VibeEdit59.5%1654.7
ModelWinTieLoss
Qwen-Image-Edit-2511731963
LongCat-Image-Edit561659
FLUX.2-klein-9B531177
FireRed-Image-Edit-1.0631474
JoyAI-Image-Edit601668
Step1X-Edit-v1p257964
VibeEdit881545

VibeEdit substantially outperforms all baselines in human preference (59.5% win rate and 1654.7 Elo), confirming that RL brings genuine visual and instruction-following gains that align with human preference rather than giving an unfair rubric advantage. Furthermore, our human audit in Table 10 confirms that the GPT-5.6-sol test judge closely aligns with real human judgments on VibeEdit outputs.

[1]. Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[2]. Gunjal, Anisha, et al. "Rubrics as rewards: Reinforcement learning beyond verifiable domains." International Conference on Learning Representations. Vol. 2026. 2026.

[3]. Liu, Jie, et al. "Flow-grpo: Training flow matching models via online rl." Advances in neural information processing systems 38 (2026): 40783-40818.

[4]. Xue, Zeyue, et al. "Dancegrpo: Unleashing grpo on visual generation." arXiv preprint arXiv:2505.07818 (2025).

[5]. Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." International Conference on Learning Representations. Vol. 2026. 2026.

W2: Fair visual comparisons with clean source images and text instructions

Thank you for this fair point. We agree that comparing VibeEdit against baselines given clean source images and text instructions is much more informative than visual-only inputs. We have added a side-by-side visual comparison across benchmark cases in Appendix D.2 (Figure 7) of the revised paper—please take a look. As shown in the figure, even with detailed text prompts, the baselines frequently edit the wrong instance among similar objects, misplace moved objects, and alter unrelated background details. These visual results align with Table 1, clearly showing why VibeEdit achieves both a higher VLM rubric score and a higher outside-region PSNR.

W3: Practical advantages of canvas instructions and multi-object editing

Thank you for being open to discussing this. Beyond multi-edit, canvas instructions offer clear practical advantages even for single edits, and VibeEdit also supports multi-edit well:

1. Advantages of on-canvas instructions

  • Over pure text prompts: Text describes what to change well, but specifying where in words is often tedious and error-prone. When several objects look alike, users have to write long positional descriptions (~21 words in our benchmark, e.g., counting objects from left to right), yet models still frequently pick the wrong instance. Text is also ill-suited for continuous spatial inputs like exact insertion spots or movement paths. Sketching a quick circle or arrow with a short note removes referential ambiguity, feels natural for artists working on tablets or touchscreens without switching between the image and a prompt box, and helps keep unedited areas intact (32.8 dB outside PSNR vs. ≤\le 24.6 dB for text baselines).
  • Over mask/brush interfaces: Mask-based editors require users to carefully paint object silhouettes, clip out-of-mask effects such as shadows or larger replacements, and cannot express source-to-target links for object movement. Rough canvas marks require no boundary tracing and handle all five edit types in one unified interface. Empirically, VibeEdit also outperforms five mask- and brush-guided editors, please see our responses to Reviewer eZ9A-W1&Q1 and Reviewer WWdX-W1&Q1 for the full breakdown.

2. Multi-edit capability

We strongly agree that multi-edit is a natural fit for canvas instructions, since placing each short note next to its mark directly ties every instruction to its target region. Following your suggestion, we evaluated VibeEdit on one-pass multi-object editing and placed the visual results in Appendix D.3 (Figure 8) of the revised paper—please take a look. As shown there, despite being trained only on single-object pairs, our model directly supports simultaneous multi-object edits (even mixing different operation types) in a single pass.

Q1: Data overlap between SFT and RL

Thank you for asking—we will make this explicit in the revised paper. The SFT and RL stages use completely disjoint datasets: while both follow the same data sources and synthesis pipeline, the 3,520 RL conditions are selected from a separate held-out pool with zero image overlap with the SFT training corpus.

Q2: Choice of GPT-5.4-mini as the RL judge

This comes down to balancing judgment precision with online training speed. Data synthesis and benchmark evaluation run offline, so we can afford higher latency and use GPT-5.6-sol for the most accurate judgments. During RL, however, each binary rubric question is queried in a separate API call, averaging about 10 questions per rollout. With 32 conditions per step across our 32 A100 GPUs and 12 rollouts per condition, every single RL step requires roughly 32×12×10≈3,84032 \times 12 \times 10 \approx 3,840 online API calls. Using GPT-5.4-mini cuts response latency substantially so the GPUs do not sit idle waiting for rewards.

Q3 (Part 1): Mask expansion for replacement data generation

After getting the binary object mask MM from SAM 3 using the selected bounding box, we dilate it outward by 16 pixels (a 33×3333 \times 33 max filter) before feeding it to FLUX.1 Fill. This margin covers thin edges of the original object so no ghosting is left behind, and gives FLUX.1 Fill a bit of extra room to form the new silhouette and blend the boundary cleanly.

Q3 (Part 2): Large size mismatches and background preservation in object replacement

Thank you for raising this question. Although we did not specifically optimize for large size gaps during training, VibeEdit still handles both small-to-large and large-to-small replacements well across many cases, as shown in Appendix D.4 (Figure 9) of the revised paper. Two factors make this possible:

  1. Soft spatial guidance at test time: VibeEdit does not apply a hard binary mask during inference; the drawn circle only acts as a soft spatial cue. When replacing a small object with a much larger one, such as turning suitcase into refrigerator, the new object can naturally extend past the original object and circle boundaries with realistic contact and shadows.
  2. Joint training with removal and addition: A replacement with a large size gap is close to removing the old object and adding a new one at its natural scale. When replacing a large object with a much smaller one, such as swapping a smartphone for a tiny coin or a cushion for a tennis ball, our joint training with object removal helps fill in the newly exposed background cleanly, while object addition guides the model to place the target at a natural size rather than stretching it to fill the old outline.

At the same time, we do observe that extreme size differences are harder and have a lower success rate than replacements between similarly sized objects. Improving robustness on such cross-scale replacements will be a direction for our future work, and we thank you again for bringing up this great question.

Reviewer_WWdXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VibeEdit uses circles, scribbles, movement arrows, and rasterized text notes to specify five types of image edits directly on a canvas. The authors construct approximately 1.55 million source–target pairs and adapt Qwen-Image-Edit using separate source/canvas conditioning, region-weighted supervised learning, and rubric-guided DiffusionNFT. On an independently collected 419-case benchmark emphasizing similar-object selection, the final model reports a rubric score of 79.9 and outside-region PSNR of 32.8 dB. Validation includes conditioning and reward ablations, human agreement with the judge on VibeEdit outputs, and user-written instruction experiments.

Critical strengths
  1. A useful interface supported by a substantial system/data contribution. Directly associating local semantic instructions with spatial marks addresses instance disambiguation in text-only editing. Covering five operations with a large training corpus makes this more than an isolated interface demonstration.

  2. A controlled ablation supports separate source/canvas conditioning. Holding the VLM input fixed, separate VAE inputs improve the rubric score from 58.6 to 67.8 and PSNR from 24.2 to 27.0 dB (Table 2). Human-label agreement and user-written instruction experiments provide additional, appropriately scoped support (Appendix A).

Critical weaknesses
  1. The main comparisons do not establish the advance over spatially informed editing. Text baselines receive a source and text; visual-only baselines receive the annotated image; VibeEdit receives a clean source and a separate canvas representation with specialized training. The complete-system result is useful, but does not isolate gains from spatial information, clean-source access, and the proposed design. Existing interactive/spatial editors are not compared on shared supported operations. The human judge audit covers only VibeEdit outputs and therefore does not independently establish the cross-method ranking. This is central to assessing the claimed comparative contribution.

  2. The RL improvement is confounded by inference settings. Base models use 30 steps and CFG 4, whereas RL models use 16 steps and CFG 1 (Section 5.1). The rubric improvement from 67.8 to 79.9 therefore compares final configurations and cannot be attributed entirely to RL. Reward-component ablations do not replace a matched-checkpoint/configuration comparison, leaving an important contribution insufficiently validated.

Questions for authors
  1. Clarification and bounded validation of the comparison: Were any strong baselines given equivalent spatial target information and a clean source reference? If not, can you evaluate a compatible baseline on the existing cases, or a spatial/interactive editor on shared supported operations? A small blinded cross-method human check would help determine whether the ranking survives the input-format differences. Results here could change my recommendation.

  2. Bounded validation of the RL claim: Can you evaluate the SFT and RL checkpoints under both reported inference configurations, at least on a representative existing subset? This would determine whether the reported RL improvement remains when step count and CFG are controlled.

5 comments

W1&Q1 (Part 1): Input fairness, comparisons with spatial editors, and cross-method human evaluation

Thank you for the constructive feedback. We ran three additional evaluations: (1) Table 1 baselines given the exact same inputs as VibeEdit, (2) spatial/interactive editors in their native formats, and (3) a blinded cross-method human study.

1. Baselines with VibeEdit's exact inputs

In Table 1, we used a single-image input across all baselines because LongCat-Image-Edit, JoyAI-Image-Edit, and Step1X-Edit-v1p2 are released for single-image conditioning, whereas Qwen-Image-Edit-2511, FLUX.2-klein-9B, and FireRed-Image-Edit-1.0 natively accept multiple reference images. To keep the visual-only setup consistent, we fed every baseline the merged annotated image A=I⊕CA = I \oplus C. Here, we adapt the three single-image pipelines to accept two VAE image latents so that all six baselines are evaluated under VibeEdit's exact multi-image setup, feeding AA to the VLM encoder and both the clean source II and decoupled canvas layer Cˉ\bar{C} to the VAE without an external text prompt:

Method (Input: I+Cˉ+AI + \bar{C} + A)Add V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
Qwen-Image-Edit-251122.2 / 17.024.5 / 14.326.4 / 16.525.4 / 16.617.2 / 17.423.1 / 16.4
LongCat-Image-Edit6.9 / 15.725.0 / 17.930.5 / 15.822.6 / 16.48.9 / 14.218.8 / 16.0
FLUX.2-klein-9B20.9 / 25.232.9 / 26.251.3 / 26.436.9 / 26.020.3 / 26.132.5 / 26.0
FireRed-Image-Edit-1.030.0 / 19.432.4 / 14.928.9 / 14.219.2 / 16.78.4 / 17.323.8 / 16.6
JoyAI-Image-Edit35.9 / 18.018.0 / 17.432.8 / 18.433.8 / 17.02.4 / 14.124.6 / 17.0
Step1X-Edit-v1p230.9 / 18.945.5 / 20.448.7 / 19.051.1 / 18.215.7 / 19.938.4 / 19.2
VibeEdit-base67.6 / 32.773.3 / 27.566.0 / 22.172.9 / 31.559.1 / 20.767.8 / 27.0
VibeEdit80.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8

Relative to the single-image visual-only setting in Table 1 (AA alone), feeding (I,Cˉ)(I, \bar{C}) brings modest gains for Qwen-Image-Edit-2511 (19.8 →\rightarrow 23.1), FireRed-Image-Edit-1.0 (22.8 →\rightarrow 23.8), and Step1X-Edit-v1p2 (37.9 →\rightarrow 38.4), while lowering scores on FLUX.2-klein-9B (37.6 →\rightarrow 32.5), LongCat-Image-Edit (23.1 →\rightarrow 18.8), and JoyAI-Image-Edit (46.4 →\rightarrow 24.6), with all six baselines remaining below 38.5 VLM.

2. Spatial and interactive editors in their native formats

To compare directly with existing spatial and interactive editors in their intended input formats, we evaluated five representative methods using clean source II, ground-truth target regions (masks, boxes, or brush strokes), and short text prompts matching our canvas notes:

MethodNative InputAdd V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
Qwen-Image-Edit-2511 (bbox)II + BBox + prompt35.5 / 17.652.1 / 20.343.4 / 16.432.2 / 17.921.8 / 17.637.0 / 17.9
BrushEditII + Mask + prompt54.5 / 31.446.6 / 28.653.0 / 32.866.5 / 31.926.9 / 27.849.5 / 30.5
MagicQuill-V2II + Brush + prompt63.3 / 17.656.5 / 17.355.7 / 15.142.6 / 17.462.1 / 16.156.0 / 16.7
MagicQuill-V1II + Brush + prompt68.6 / 32.850.9 / 33.263.6 / 30.966.8 / 29.144.5 / 32.658.9 / 31.7
FLUX.1-FillII + Mask + prompt67.7 / 32.559.2 / 33.470.7 / 33.472.9 / 32.750.0 / 29.164.1 / 32.2
VibeEdit-baseCanvas (I,C)(I, C)67.6 / 32.773.3 / 27.566.0 / 22.172.9 / 31.559.1 / 20.767.8 / 27.0
VibeEditCanvas (I,C)(I, C)80.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8

Even with ground-truth target regions and clean source access, all five baselines trail VibeEdit-base (67.8) and VibeEdit (79.9) overall, and VibeEdit leads on almost every individual operation.

Continued in W1&Q1 (Part 2), which reports the blinded cross-method human evaluation.

W1&Q1 (Part 2): Input fairness, comparisons with spatial editors, and cross-method human evaluation

3. Blinded cross-method human evaluation

We conducted a blind user study with 10 human raters over 500 pairwise comparisons, evaluating VibeEdit and the six open-source baselines from Table 1 in their stronger text-instructed setting. Raters judged instruction following and visual quality directly without seeing our rubric. We report strict win rate, Elo score (ties counted as 0.5), and the win/tie/loss match counts for each model:

ModelWin RateElo
Qwen-Image-Edit-251147.1%1543.8
LongCat-Image-Edit42.7%1525.8
FLUX.2-klein-9B37.6%1420.0
FireRed-Image-Edit-1.041.7%1506.5
JoyAI-Image-Edit41.7%1413.6
Step1X-Edit-v1p243.8%1435.5
VibeEdit59.5%1654.7
ModelWinTieLoss
Qwen-Image-Edit-2511731963
LongCat-Image-Edit561659
FLUX.2-klein-9B531177
FireRed-Image-Edit-1.0631474
JoyAI-Image-Edit601668
Step1X-Edit-v1p257964
VibeEdit881545

VibeEdit clearly outperforms all baselines on both win rate (59.5%) and Elo score (1654.7), confirming that VibeEdit's advantage holds in blinded human judgment. Furthermore, our human audit in Table 10 confirms that the GPT-5.6-sol test judge closely aligns with real human judgments on VibeEdit outputs.

W2&Q2: Comparing SFT and RL under identical inference settings

Thank you for raising this point. We evaluated the two models under different inference settings because their training regimes operate under distinct timestep and conditioning setups. Specifically, VibeEdit-base is trained via standard SFT across all 1,000 timesteps, randomly dropping the VLM condition to learn an unconditional branch for classifier-free guidance. At inference, we therefore follow the default Qwen-Image-Edit setup of 30 denoising steps and CFG = 4. In contrast, during RL post-training (Section 5.1), rollouts are generated in just 8 steps with CFG = 1 (without condition dropout), and the policy is optimized directly on these few-step, guidance-free trajectories. Because the two checkpoints are trained under fundamentally different sampling regimes, we evaluate VibeEdit at 16 steps and CFG = 1 to match its RL training regime while cutting forward passes nearly fourfold (16 single passes vs. 30 two-pass CFG steps).

To isolate the effect of RL from inference settings, we evaluated both checkpoints under both configurations on the full 419-case benchmark:

MethodAdd V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
VibeEdit-base · 30 steps / CFG 467.6 / 32.773.3 / 27.566.0 / 22.172.9 / 31.559.1 / 20.767.8 / 27.0
VibeEdit-base · 16 steps / CFG 164.7 / 33.172.9 / 28.068.0 / 22.272.8 / 31.758.5 / 21.167.4 / 27.4
VibeEdit · 30 steps / CFG 480.4 / 34.785.2 / 35.476.6 / 30.979.2 / 34.578.8 / 27.580.0 / 32.6
VibeEdit · 16 steps / CFG 180.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8

Holding the inference setting fixed, RL improves VibeEdit-base from 67.8 to 80.0 VLM (27.0 to 32.6 dB) at 30 steps / CFG 4, and from 67.4 to 79.9 VLM (27.4 to 32.8 dB) at 16 steps / CFG 1. By contrast, switching inference settings moves either model by no more than 0.6 VLM and 0.4 dB.

We also ran step and CFG sweeps for VibeEdit on the full benchmark:

Steps (CFG = 1)Add V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
879.5 / 34.784.4 / 35.878.2 / 31.178.9 / 34.975.4 / 27.179.3 / 32.7
1680.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8
2080.3 / 34.785.8 / 35.475.2 / 31.381.6 / 34.676.3 / 27.679.8 / 32.7
3079.5 / 34.582.9 / 35.378.6 / 31.280.9 / 34.575.8 / 27.379.5 / 32.6
CFG (16 steps)Add V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
180.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8
280.6 / 34.685.2 / 36.177.7 / 31.280.4 / 35.376.9 / 27.580.2 / 32.9
480.3 / 34.485.5 / 35.576.8 / 31.180.0 / 35.676.2 / 27.279.8 / 32.8
680.4 / 34.784.4 / 35.576.8 / 31.079.2 / 34.676.0 / 27.179.4 / 32.6

Scores stay within 79.3–79.9 VLM across 8–30 steps and 79.4–80.2 VLM across CFG 1–6, so CFG = 1 achieves near-optimal results without doubling forward passes. We will include these comparisons in the revised paper.

Reviewer_eZ9AOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VibeEdit, an image editing framework that enables users to specify editing intentions directly on images through spatial annotations and optional text notes, without requiring separate text prompts. The framework combines layer-decoupled conditioning, region-weighted supervised fine-tuning, and rubric-guided reinforcement learning to support five editing operations while preserving unedited image regions.

Critical strengths
  1. The proposed canvas-based interaction provides an intuitive and practical approach to image editing, particularly for accurately identifying target objects in complex scenes with similar instances.
  2. Extensive experiments demonstrate strong performance over existing baselines in both editing accuracy and background preservation, supported by a large-scale training dataset and comprehensive ablation studies.
Critical weaknesses
  1. The paper lacks controlled comparisons with spatially guided editing methods and does not isolate the benefits of canvas instructions from task-specific training and reinforcement learning.
  2. The RL rewards closely overlap with the evaluation metrics, raising concerns about possible metric overfitting and the reliability of the reported improvements.
  3. The proposed framework primarily combines existing techniques, including region-weighted supervision and rubric-guided RL, while the novelty and effectiveness of its individual components remain insufficiently demonstrated.
Questions for authors
  1. Could the authors clarify how the benefits of canvas instructions are distinguished from those of task-specific training and RL? Are there any controlled comparisons under similar training conditions?
  2. Given the overlap between the RL rewards and evaluation metrics, could the authors provide further evidence that the improvements generalize beyond the optimized criteria, such as independent human evaluations?
  3. Could the authors further clarify the methodological contributions beyond existing techniques and provide additional insights into the effectiveness of the proposed components?
3 comments

W1&Q1: Controlled comparisons and disentangling canvas instructions from SFT and RL

Thank you for raising this point. We address both parts of the question below by (1) comparing VibeEdit with five spatially guided editing baselines given ground-truth target regions, and (2) analyzing how canvas conditioning, SFT, and RL each contribute under controlled settings on the same backbone.

(1) Controlled Comparison with Spatially Guided Baselines

In our main paper (Table 1), baselines received either the annotated image AA (Visual-only) or a detailed external text prompt describing the target location (Text-instructed). To ensure a fair comparison where baselines also receive explicit spatial grounding, we evaluated five spatially guided methods on the VibeEdit benchmark by providing them with ground-truth target regions (masks, bounding boxes, or brush strokes) paired with text prompts (matching the concise semantic notes used in VibeEdit):

MethodGuidance FormatAdd V/PRemove V/PAttribute V/PReplace V/PMove V/POverall V/P
Qwen-Image-Edit-2511 (bbox)BBox + prompt35.5 / 17.652.1 / 20.343.4 / 16.432.2 / 17.921.8 / 17.637.0 / 17.9
BrushEditMask + prompt54.5 / 31.446.6 / 28.653.0 / 32.866.5 / 31.926.9 / 27.849.5 / 30.5
MagicQuill-V2Brush + prompt63.3 / 17.656.5 / 17.355.7 / 15.142.6 / 17.462.1 / 16.156.0 / 16.7
MagicQuill-V1Brush + prompt68.6 / 32.850.9 / 33.263.6 / 30.966.8 / 29.144.5 / 32.658.9 / 31.7
FLUX.1-FillMask + prompt67.7 / 32.559.2 / 33.470.7 / 33.472.9 / 32.750.0 / 29.164.1 / 32.2
VibeEdit-baseCanvas Instruction (I,C)(I,C)67.6 / 32.773.3 / 27.566.0 / 22.172.9 / 31.559.1 / 20.767.8 / 27.0
VibeEditCanvas Instruction (I,C)(I,C)80.3 / 34.785.4 / 35.877.4 / 31.279.9 / 34.976.4 / 27.179.9 / 32.8

Even without RL, VibeEdit-base (67.8 VLM) already outperforms all five spatially guided baselines, while VibeEdit achieves the best overall results on both metrics (79.9 VLM / 32.8 dB). Unlike static masks or brushes that only mark isolated regions and struggle with relational edits like Move, canvas instructions naturally unify localization, directional arrows, and short notes in one interface.

(2) Isolating Canvas Instructions from SFT and RL

To separate the effect of canvas instructions from SFT and RL, we compare controlled variants on the same Qwen-Image-Edit backbone:

Condition FormatSFTRLVLMPSNR
Text Prompt✓59.517.4
Canvas Instruction19.816.3
Canvas Instruction✓67.827.0
Canvas Instruction✓✓79.932.8

Even after fine-tuning on our VibeEdit dataset with text prompts, the text-conditioned baseline only reaches 59.5 VLM and 17.4 dB, as text-only supervision cannot precisely pin down target instances or edit boundaries and introduces ambiguity when multiple similar objects are present. Training on the same dataset with canvas instructions instead reaches 67.8 VLM and 27.0 dB, substantially outperforming the SFT text baseline under identical training conditions. Consistent with Table 2 in the main paper, where removing the VAE canvas layer under the same SFT setup drops the score from 67.8 to 24.7, this confirms that the improvement comes from explicit spatial grounding on the canvas rather than task-specific fine-tuning alone. Meanwhile, SFT bridges the initial domain gap for unseen canvas marks (19.8 to 67.8), and RL brings a further gain to 79.9 VLM and 32.8 dB.

W2&Q2: Reward–evaluation overlap and independent human validation

Thank you for this thoughtful question. Because SFT relies on a proxy objective (flow-matching MSE) that does not directly measure instruction following or visual fidelity, RL post-training is intended to align the model with what human users actually care about at test time. For this reason, using RL rewards that mirror target evaluation criteria to align with human preference is standard practice in both LLM and diffusion RL literature. For example, in LLMs, DeepSeek-R1[1] optimizes rule-based accuracy rewards evaluated on math and coding benchmarks, and Rubrics as Rewards[2] optimizes LLM-judged checklists evaluated on rubric benchmarks like HealthBench; in visual generation, Flow-GRPO[3], DanceGRPO[4], and DiffusionNFT[5] directly optimize target evaluation metrics such as GenEval, OCR, HPSv2.1, and PickScore. Similarly, in localized image editing, human preference centers on completing the requested edit, preserving unedited regions, and avoiding local artifacts. Designing both our RL reward and benchmark rubric around these three aspects (using a lighter GPT-5.4-mini judge during RL and GPT-5.6-sol at test time, alongside outside-region PSNR) directly steers the model toward human-preferred edits.

Following your suggestion, we also conducted a blind user study with 10 human raters over 500 pairwise comparisons, evaluating VibeEdit alongside all six open-source baselines from Table 1 in their stronger text-instructed setting. Human raters were not shown our rubric questions and judged instruction following and visual quality directly:

ModelWin RateElo
Qwen-Image-Edit-251147.1%1543.8
LongCat-Image-Edit42.7%1525.8
FLUX.2-klein-9B37.6%1420.0
FireRed-Image-Edit-1.041.7%1506.5
JoyAI-Image-Edit41.7%1413.6
Step1X-Edit-v1p243.8%1435.5
VibeEdit59.5%1654.7
ModelWinTieLoss
Qwen-Image-Edit-2511731963
LongCat-Image-Edit561659
FLUX.2-klein-9B531177
FireRed-Image-Edit-1.0631474
JoyAI-Image-Edit601668
Step1X-Edit-v1p257964
VibeEdit881545

VibeEdit substantially outperforms all baselines in human preference (59.5% win rate and 1654.7 Elo), confirming that RL brings genuine visual and instruction-following gains that generalize to human preference rather than overfitting the reward judge. Furthermore, our human audit in Table 10 confirms that the GPT-5.6-sol test judge closely aligns with real human judgments on VibeEdit outputs.

[1]. Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

[2]. Gunjal, Anisha, et al. "Rubrics as rewards: Reinforcement learning beyond verifiable domains." International Conference on Learning Representations. Vol. 2026. 2026.

[3]. Liu, Jie, et al. "Flow-grpo: Training flow matching models via online rl." Advances in neural information processing systems 38 (2026): 40783-40818.

[4]. Xue, Zeyue, et al. "Dancegrpo: Unleashing grpo on visual generation." arXiv preprint arXiv:2505.07818 (2025).

[5]. Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." International Conference on Learning Representations. Vol. 2026. 2026.

W3&Q3: Methodological novelty and component effectiveness

We thank the reviewer for this comment. Our main goal is not to propose isolated network changes on a standard editing setup, but to introduce canvas instructions as a new way of interacting with image editors, letting users combine rough spatial marks, movement arrows, and short handwritten notes directly on the image without separate text prompts or careful mask painting. Making this interface work in practice required building the full data and evaluation stack from the ground up. Specifically, we designed a multi-stage generation, segmentation, and VLM-filtering pipeline to construct 1.55M clean edit pairs across five operations, and manually collected a 419-case benchmark focused on multi-object scenes with human-verified evaluation.

Methodologically, canvas editing poses challenges that standard editors do not face: the model has to read drawn marks without copying them into the output, handle rough strokes and out-of-object text notes, and edit local regions while keeping the rest of the image intact. Directly applying existing methods degrades performance, and each of our four design choices is backed by controlled ablations:

  1. Layer-decoupled conditioning. Feeding a single image (II or AA) to the VAE either drops spatial layout (24.7 VLM) or leaves visible strokes in the result (58.6 VLM / 24.2 dB). Sending the annotated image AA to the VLM while encoding the clean source II and the gray-background instruction layer Cˉ\bar{C} separately in the VAE removes stroke artifacts and raises VibeEdit-base to 67.8 VLM and 27.0 dB (Table 2).
  2. Dilated object-and-annotation SFT. Conventional region-weighted losses only upweight the object mask MobjM_{\mathrm{obj}}. Because text notes and arrows often sit outside the target object, we instead supervise the union Mobj∨MannM_{\mathrm{obj}} \lor M_{\mathrm{ann}}. Even so, weighting without dilation lowers the score from 66.2 to 64.2 (Table 3) as user circles only loosely bound the object; dilating this combined mask accommodates rough strokes and boundary transitions, raising the score to 67.8.
  3. RL for local editing. Standard RubricRL is built for full-image T2I generation, whereas local editing also needs to keep the background unchanged. Rewarding only edit success and local quality hurts unedited areas, dropping PSNR from 28.6 to 28.0 dB (Table 5). We address this by giving the judge aligned local crops and adding outside-preservation rubrics with a PSNR penalty. Combined with per-task reward normalization and high-variance data selection (79.9 vs. 76.6 for low-variance, Table 4), RL improves VibeEdit from 67.8 / 27.0 dB to 79.9 / 32.8 dB.
  4. Offline pair storage with online rendering. Saving clean image pairs and masks offline while rendering diverse strokes, bidirectional edits, and multiple fonts on the fly enables positive transfer across the five edit types (Table 6), effective deletion cues (Table 9), and reliable generalization to real handwriting and unseen stroke colors (Tables 7, 8).

Comments on the paper

2 comments

Thank you for taking the time to review our rebuttal and the new experiments. We are glad our responses addressed your concerns, and we truly appreciate your willingness to reconsider your score and support our work, which means a lot to us. Many thanks again for your helpful feedback and support.

Dear Reviewer,

Thank you very much for your time and for carefully considering our rebuttal. You previously mentioned that our responses had satisfactorily addressed your main concerns and kindly raised your score from 2 to 4. We sincerely appreciate your reconsideration.

However, I just noticed that the score has now been lowered back to 2 and your previous comment has been deleted. Since the rebuttal period is closing in just a few hours, I was wondering whether this might have been an accidental change.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.