acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Scribble-Guided Image Editing: A Progressive Supervision Curriculum for Coherent Multi-Task Editing

Abstract

Scribble-guided image editing combines text with coarse strokes to specify both what to edit and where. However, current image editors often fail to follow scribble instructions, especially for multiple requests. We diagnose two gaps in the required training supervision. First, precise supervision and visually coherent supervision play complementary roles: the former supports reliable local editing, while the latter improves visual coherence with the source image. Second, exposure to multiple scribbles alone is insufficient for multi-task editing; complete multi-task supervision is needed for reliable binding between scribbles, instructions, and edits. Based on these findings, we propose a Progressive Supervision Curriculum. Layered Synthetic Data provide precise supervision for single-task edits. Multi-Task Mosaicking extends these samples into complete multi-task tuples. Visually Coherent Synthetic Data provide supervision for visual coherence. Stage I learns single-task editing and multi-task binding from Layered Synthetic Data and mosaicked tuples. Stage II refines visual coherence with Visually Coherent Synthetic Data. We render scribbles directly into the input image and train only a LoRA adapter. On VIBE, our method achieves the best overall single-task result and reaches performance comparable to closed-source models on multi-task editing. We will release our dataset and model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scribble-guided image editing, progressive supervision, multi-task binding, visual coherence, synthetic data construction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_68CfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper combines precise layered synthetic edits, multi-task mosaicking, and subsequent visual-coherence refinement. LoRA adaptation improves scribble-guided editing on VIBE, with additional results on a second backbone.
Critical strengths
  • The distractor-scribble control usefully distinguishes exposure to multiple marks from supervision for multiple requested edits.
  • Large improvements over the base model are supported by component ablations, a second backbone, and a small human study.
Critical weaknesses
  • The experiments demonstrate useful data sources, but do not isolate the benefit of their proposed training order from simply adding more data and training.
  • Equal numbers of training examples need not imply equal supervision: mosaicked examples contain more source images and requested edits.
Questions for authors
  1. Evidence request: Is there a comparison with mixed or reversed-order training using the same data exposure and training budget? This would establish whether the curriculum order itself matters.
  2. Clarification: In the supervision comparisons, how were source-image exposure, supervised edit counts, and training compute controlled across single-task and mosaicked examples?
3 comments
AuthorsOpenReview ↗

Title: Response to Reviewer 68Cf (1/1)

W1 & Q1: Effect of the training order

Thank you for this suggestion. We agree that the original experiments did not isolate the benefit of stage ordering from that of additional data and training. We have therefore added two complementary controls. All configurations use the same datasets, a total of 4,500 training steps, and a global batch size of 64.

  • Reversed-order training: We first train on Visually Coherent Synthetic Data for 500 steps, followed by Layered Synthetic Data with Multi-Task Mosaicking for 4,000 steps. This preserves the number of training updates allocated to each data source while reversing their order.
  • Mixed single-stage training: We jointly sample from both datasets throughout training, testing Layered : Visually Coherent ratios of 40:1, 10:1, 5:1, and 1:1. These experiments evaluate different sampling balances under the same total step budget; exposure to each data source varies with the mixing ratio.
TrainingIA (1-Task)VC (1-Task)
Mixed (40:1)89.6491.11
Mixed (10:1)89.1190.97
Mixed (5:1)88.1190.67
Mixed (1:1)90.4791.78
Reversed order88.6788.47
Ours (Stage I → Stage II)90.1693.81

Reversing the stages reduces VC from 93.81 to 88.47, a 5.34-point difference, despite retaining the same per-source training-step allocations. This provides evidence that the stage order affects performance under the same data and training budget.

The mixed-training results provide complementary evidence: our schedule achieves 2.03 points higher VC than the strongest tested mixture (1:1), with similar IA (90.16 vs. 90.47). Overall, the proposed schedule achieves the highest VC among the tested configurations, with the clearest benefit in visual coherence, consistent with the purpose of Stage II.

W2 & Q2: Supervision exposure in the Sec. 3.2 comparison

Thank you for pointing this out. We agree that equal numbers of training examples do not imply equal amounts of supervision. In Sec. 3.2 (Figure 3), “equal-size” refers to 180K training examples per dataset. All three settings use 4,000 optimization steps and a global batch size of 64, corresponding to 256K training-example presentations:

  • Single-task: 180K examples, each containing one source image and one supervised edit.
  • Distractor-scribble: The same 180K single-task examples augmented with instruction-unrelated scribbles. Each example still contains one source image and one supervised edit.
  • Multi-task: 144K single-task examples and 36K mosaicked examples, with 18K two-tile and 18K four-tile mosaics. Each tile contributes one source image and one supervised edit, giving an average of 1.4 source-image instances and 1.4 supervised edits per training example.

Under this sampling mixture, the single-task and distractor-scribble settings each receive 256K source-image exposures and 256K supervised-edit exposures, while the multi-task setting receives approximately 358.4K of each. These counts include repeated presentations and should not be interpreted as numbers of distinct source images.

The original comparison therefore evaluates different supervision formats under matched training-example and optimization-step budgets, but does not match cumulative edit exposure. To address this difference, we additionally extend single-task training to 6,000 steps. This increases its cumulative edit exposure to 384K, exceeding that of the multi-task setting:

Training dataStepsSupervised edits1-Task2-Task3-Task
Single-task4,000256K87.4968.1650.21
Single-task6,000384K87.4769.1257.41
Multi-task4,000~358.4K85.0979.0170.26

Longer single-task training improves 3-Task performance by 7.20 points, confirming that additional training does help. Nevertheless, the multi-task setting remains 9.89 points higher on 2-Task and 12.85 points higher on 3-Task, despite using fewer optimization steps and fewer cumulative edit exposures. This multi-task advantage comes with a lower 1-Task score in this comparison (85.09 vs. 87.47).

Thus, increasing single-task exposure and training steps alone does not reproduce the observed multi-task gains. These results support the value of training on jointly specified scribbles, instructions, and edit targets, beyond simply increasing the cumulative number of supervised edits.

Solved my concerns, so I will keep my rating.

Thank you for confirming that our responses addressed your concerns and for supporting our work. We will incorporate the additional experiments and clarifications from the rebuttal into the revision.

Reviewer_xHRxOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a multi-stage supervision pipeline for scribble-guided complex image editing. To improve base editing models on scribble and text conditioned editing tasks, the authors introduce a dataset generation pipeline with three components: Layered Synthetic Data, which targets single-task editing performance; Visually Coherent Data, which provides supervision for visual coherence; and Multi-Task Mosaicking, which provides explicit supervision for editing with multiple instructions and scribbles. They also propose a curriculum strategy that trains on these data types progressively. On the recently introduced VIBE benchmark, they show empirical improvements over a base editing model, closing the gap to closed-source methods.

Critical strengths
  1. The proposed data generation pipeline appears largely automatic and therefore scalable, which could make it useful beyond this specific work.

  2. The improvements on the VIBE benchmark appear substantial for both base models, Qwen (Tab. 2) and Flux.2-Dev (Tab. 6).

Critical weaknesses
  1. The paper’s technical contribution is limited. The data generation pipeline, which is the core of the approach, is fairly straightforward and consists of simple chaining of existing foundation models: VLM-based instruction generation, object segmentation, and editing with existing models, followed by simple grid stitching for “multi-task supervision.”

  2. The proposed fine-tuning scheme is evaluated on only two base models (Qwen-Image-Edit in the main paper and Flux.2-Dev in the appendix), and on a single, recently introduced benchmark (VIBE) that relies on LLM-as-a-judge scoring. No standard editing metrics are reported, such as CLIP similarity for text alignment or masked reconstruction metrics for background preservation. In addition, the paper does not show that the fine-tuned models retain their general editing capabilities on standard editing benchmarks.

Questions for authors

Please address the concerns above. In addition:

  1. What exactly is the problem with the “collage-like data” shown in Figure 2(a), and what distinguishes the Visually Coherent Data from it?

  2. The numbers in Table 1 lack context. It is unclear whether scores around 90 on these metrics are good or bad, and what a one-point difference signifies. Could you elaborate?

  3. What is the purpose of Section 3.2? It shows that a hypothetical “distractor-scribble supervision” performs worse than “complete multi-task supervision” on multi-task editing. Isn’t this result expected?

  4. Please expand on the Layered Synthetic Data generation approach (Section 4.2). What do the layers "l" represent, and what exactly is the proposed method?

  5. The Visually Coherent Synthetic Data pipeline appears automatic and scalable, apart from standard curation, which could likely also be automated with a modern VLM. Why do you describe it as “difficult to scale,” and why is the Layered Synthetic Data necessary?

  6. Were the editing baselines exposed to scribble-based editing pairs during training, or are they evaluated in a zero-shot, domain-shifted setting?

  7. Closed-source models already perform largely on par with the proposed approach out of the box. Why do you consider scribble-based editing to still be a relevant problem?

7 comments
AuthorsOpenReview ↗

Title: Response to Reviewer xHRx (1/5)

W1: Limited technical contribution

We agree that the individual components use established techniques. Our contribution is a task-specific supervision construction and progressive training scheme for precise, visually coherent, and multi-task scribble-guided editing, supported by comparisons that test the proposed design choices. The complete scheme raises Qwen-Image-Edit-2511’s VIBE Deictic-Level 1-Task score from 63.26 to 89.23, an improvement the reviewer also describes as “substantial.” It achieves the highest single-task score among the evaluated models and multi-task performance comparable to strong closed-source models.

To our knowledge, our planned release would provide the first public and complete pipeline for both single-task and multi-task scribble-guided editing, comprising the training dataset, data construction and training code, and model weights. The technical contributions and supporting evidence are as follows:

1. Each design exploits the properties of scribble-guided editing, and its value is supported by controlled comparisons with simpler alternatives.

  • Precise supervision from layered image structure. Scribbles mark local objects, so we construct supervision from layered images, where objects correspond to RGBA layers. Editing and re-rendering a layer yields an exact target image and edit-region mask without annotation, and the mask is reused for edit-region weighting. Stage I alone raises the 1-Task score from 63.26 to 88.10, and edit-region weighting improves 1/2/3-Task scores from 87.41/81.90/71.72 to 88.10/82.65/73.29 (Table 3).
  • Complementary supervision sources and progressive training. Sec. 3.1 shows that scalable precise supervision gives comparable instruction adherence, while visually coherent data mainly improve visual coherence (Table 1). We therefore learn accurate editing first and then refine visual coherence with a small curated set; Stage II improves VC by about four points across 1/2/3-Task (Table 4). The additional schedule ablations (see our response to Reviewer 68Cf, Q1) further support this order: under the same data and step budget, our schedule reaches a 1-Task VC of 93.81, compared with 91.78 for the best mixture and 88.47 for reversed-order training.
  • Complete supervision for jointly executing multiple edits. The distractor-scribble control in Sec. 3.2 rules out the inexpensive alternative of only exposing the model to multiple scribbles (Figure 3). As noted by Reviewer 68Cf, this control “usefully distinguishes exposure to multiple marks from supervision for multiple requested edits.” We therefore construct complete multi-task supervision with Multi-Task Mosaicking, which raises 2/3-Task scores on VIBE’s natural multi-task examples from 73.21/49.67 to 82.65/73.29 (Table 3). The additional exposure control (see our response to Reviewer 68Cf, Q2) shows that this gain does not come only from more supervision: single-task training with more cumulative edit exposures (384K vs. ~358.4K) still scores 9.89/12.85 points lower on 2/3-Task.

2. The data construction is low-cost, largely automatic, and scalable.

  • As the reviewer notes, the pipeline is “largely automatic and therefore scalable.” For layered data, target images and masks come directly from layer operations, scribbles are synthesized from templates, and instructions are generated by an LLM, without manual annotation.
  • The progressive schedule places the main training load on this scalable data (180K examples) and uses only 4.5K visually coherent examples, which are relatively more costly because they require editing-model generation and human curation (see Q5).
  • Mosaicking builds multi-task supervision by reusing existing single-task pairs. It requires no additional multi-task data collection and avoids the conflicts between edits that can arise when multi-task targets are constructed through multi-turn editing. As noted by Reviewer a8PY, this design is “elegant and intuitive.”

3. Demonstrated applicability across base editing models.

  • Scribbles are rendered into the input image, preserving the standard image-and-text editing interface. Applying the same scheme to two additional base models produces the following improvements:
Base model1-Task: Base → Ours3-Task: Base → Ours
FLUX.2-Dev (Table 6)33.43 → 81.5318.36 → 52.05
Qwen-Image-2.1 (additional experiments)67.70 → 87.5360.75 → 77.33

These results support applicability beyond the primary model. The planned complete release will enable independent reproduction and adaptation, while the supervision comparisons and construction recipes can also inform future closed-source model development by providing concrete, experimentally tested design choices for further investigation.

AuthorsOpenReview ↗

Title: Response to Reviewer xHRx (2/5)

W2: Evaluation scope and general editing ability

Thank you for these suggestions. We have added general-editing evaluations with the scribble LoRA enabled and clarify our choice of evaluation metrics below.

1. General editing performance after scribble fine-tuning. We compare our fine-tuned model with the original Qwen-Image-Edit-2511 on GEdit-Bench and ImgEdit-Bench. The LoRA remains enabled throughout these evaluations.

GEdit-Bench (averaged over 11 task types; SC: Semantic Consistency, PQ: Perceptual Quality, O: Overall)

ModelEN SCEN PQEN OCN SCCN PQCN O
Baseline8.9687.8528.2408.8847.8318.195
Ours9.0167.8368.2808.9057.7808.142
Δ+0.048−0.016+0.040+0.021−0.051−0.053

ImgEdit-Bench (nine task types; Overall is the category average)

ModelExtractAddRemoveAdjustReplaceBackgroundStyleComposeActionOverall
Baseline4.134.654.514.464.534.314.733.744.714.42
Ours4.194.674.624.524.594.224.793.964.644.47
Δ+0.06+0.02+0.11+0.06+0.06−0.09+0.06+0.22−0.07+0.05

The overall scores remain close to those of the base model on both benchmarks, with gains in some dimensions and declines in others. These results suggest that scribble fine-tuning largely preserves general editing performance in the evaluated settings, even with the adapter enabled. Since our method updates only the LoRA parameters, the adapter can also be disabled to use the unchanged base model.

2. Choice of metrics for scribble-guided editing. CLIP similarity and masked reconstruction metrics can provide useful supplementary information, but neither directly captures all the requirements of scribble-guided editing:

  • CLIP similarity measures broad text–image alignment, whereas our instructions specify the intended edit jointly through text and scribbles. Global CLIP similarity does not explicitly assess whether the edit targets the object indicated by the scribble—for example, whether the correct instance is edited when several objects belong to the same category.
  • Masked reconstruction metrics are useful for measuring preservation when the unaffected regions can be reliably identified. However, a valid edit can change regions beyond the target object’s mask, such as its cast shadow or surrounding occlusion boundaries. A mask that excludes these legitimate changes may therefore penalize a visually coherent result.

We consequently use VIBE as our primary task-specific evaluation. It separately assesses Instruction Adherence, Contextual Preservation, and Visual Coherence, covering correct execution of the requested edit, preservation of non-target content, and integration with the surrounding scene. The VIBE paper (https://arxiv.org/abs/2602.01851) reports high correlation between its evaluator and human expert scores. Our human evaluation (Figure 5) provides complementary evidence, with overall findings consistent with the VIBE results.

Q1: Collage-like vs. visually coherent data

Thank you for asking us to clarify this distinction. The limitation of the collage-like data in Figure 2(a) is that precise spatial edits do not necessarily produce visually coherent results.

  • Collage-like data: In our layered construction, edits are performed through explicit layer operations. For example, moving an object preserves the appearance encoded in its layer, without automatically adapting its lighting, cast shadow, or perspective to the destination. The existing layer order may also produce inappropriate occlusion relationships at the new location. These edits can therefore appear unnatural, even when the correct object is moved to the requested position. Nevertheless, the layer operations provide precise spatial supervision and allow the data to be generated automatically at scale.
  • Visually coherent data: We use an image editing model to generate candidate edits and human curation to select results that integrate naturally with the surrounding scene. This provides supervision for scene-dependent effects such as lighting, shadows, and object–background blending, at a higher generation and curation cost.

Table 1 reports slightly higher mean IA for training with collage-like data and higher mean VC for training with visually coherent data. These observations motivate using the two sources together: scalable, precise supervision for learning localized edits, followed by visually coherent supervision for refining visual integration.

To make this distinction clearer, we will revise Figure 2 to show more representative source–target training pairs for both data types and highlight the relevant differences in visual integration.

AuthorsOpenReview ↗

Title: Response to Reviewer xHRx (3/5)

Q2: Context for the scores in Table 1

Thank you for highlighting the need for clearer context. Table 1 reports VIBE’s Instruction Adherence (IA) and Visual Coherence (VC) sub-scores. We clarify their scale and the interpretation of the observed differences below.

1. What do these scores measure? Both sub-scores are reported on a 0–100 scale. IA averages three binary checks: correct scribble localization, operation type, and textual semantics. VC averages checks for style consistency, visual integration, and absence of artifacts; following VIBE, VC is set to zero when IA is zero. Each Deictic task contains 100 human-annotated samples, and the scores are averaged across samples and three independent judge evaluations.

Thus, one point represents an absolute difference of 0.01 in the mean normalized sub-score. A score of 90 should not be interpreted as 90% of images having fully successful edits.

2. What performance level does a score around 90 represent? For reference, the original Qwen-Image-Edit-2511 achieves 1-Task IA/VC of 68.78/71.83, while GPT-Image-2 achieves 88.98/92.75. Scores around 90 therefore place the models in Table 1 well above the base model and close to a strong closed-source baseline on these evaluated criteria.

3. How should the differences in Table 1 be interpreted? We agree that the aggregate differences are modest: the two supervision sources achieve similar IA, and their mean 1-Task VC differs by only 0.97 points. We treat these as descriptive trends and do not claim statistical significance from the reported averages alone.

The aggregate also includes near-ceiling tasks, such as Removal, whose VC is approximately 99. The breakdown below provides additional context, showing larger observed VC differences for Addition and multi-task editing:

Training dataIA (1T)VC (1T)VC (Addition)VC (2T)VC (3T)
Collage-like88.5989.2889.7878.0263.70
Visually coherent87.1690.2596.4481.4868.15
Δ (Visually coherent − Collage-like)−1.43+0.97+6.66+3.46+4.45

These results motivate combining the two supervision sources: scalable layered data provide strong instruction adherence, while visually coherent data offer additional benefits for visual integration. Table 4 provides further evidence consistent with this role: after Stage II refinement, VC increases by approximately four points across the 1/2/3-Task settings.

Q3: Purpose of Sec. 3.2

We agree that complete multi-task supervision is expected to outperform distractor-scribble supervision. The purpose of Sec. 3.2 is to determine whether the gap is small enough for distractor augmentation to serve as a practical, low-cost substitute.

Directly generating natural multi-task training pairs can be costly: multiple scribbles and instructions must be aligned with an output that correctly executes all requested edits. Distractor augmentation, by comparison, simply adds unrelated scribbles to existing single-task samples without requiring new edited targets. It is therefore useful to test how much multi-task performance this inexpensive augmentation can recover.

Figure 3 reports the following results:

Supervision2-Task3-Task
Single-task68.1650.21
Distractor-scribble71.1256.46
Complete multi-task79.0170.26

Distractor augmentation improves multi-task performance, but complete multi-task supervision provides a further 7.89-point gain on 2-Task and 13.80-point gain on 3-Task. Thus, exposure to multiple scribbles alone does not recover the performance achieved with complete multi-task targets in our setting.

This comparison supports explicitly supervising the joint execution of multiple requested edits and motivates Multi-Task Mosaicking: an inexpensive way to construct complete scribble–instruction–edit supervision by reusing existing single-task pairs.

Reviewer_a8PYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the challenges of scribble-guided image editing. It proposes a framework for generating synthetic data to enable precise and coherent scribble-guided editing. It then introduces a progressive training framework built on the proposed dataset, incorporating multi-task mosaicking and an edit-focused loss, and demonstrates promising results with open-source models compared to closed-source alternatives.

Critical strengths
  • The Multi-Task Mosaicking trick is elegant and intuitive. A straightforward alternative would be to consider multi-turn editing to construct multi-task training samples. However, as the authors also mention, this could introduce conflicts and interference between edits and scribbles and is hard to scale. The proposed mosaicking approach is a neat way to avoid these issues.
  • The paper includes a human evaluation (Fig. 5) comparing the proposed method against closed-source models, which adds value to the experimental evaluation.
  • The motivation, analysis, and methodology are clearly presented. Overall, the paper is well-written and easy to follow.
Critical weaknesses
  • While the Multi-Task Mosaicking approach is neat, the resulting training samples seem somewhat out of distribution compared to actual multi-editing scenarios. I would suggest incorporating even a small curated set of real-world multi-editing samples, potentially constructed through multi-turn image editing while ensuring that the edits do not interfere with each other (using a VLM or manual verification). I believe such an additional training signal, even at a small scale, could be highly beneficial.
  • In the dataset construction pipeline, the authors mainly consider adding or removing objects. I assume the same pipeline could be extended to replacing or swapping objects. However, what about moving an object? I do not see this task explicitly considered during data generation or training. How does the model manage to perform this task and improve upon the baselines (e.g., Table 2, column TR, or Fig. 1, where the air conditioner is moved)?
  • An additional experiment demonstrating the effectiveness of progressive training would be valuable. Specifically, it would be useful to compare against simply training on a mixture of both datasets (precision and coherence) and show that this approach underperforms. If the concern is an imbalance between the datasets, this could potentially be addressed by adjusting the sampling weights of each subset. I believe such an ablation is necessary to establish whether the progressive training strategy itself provides a meaningful benefit.
  • The overall methodological novelty seems somewhat limited. Constructing synthetic datasets for scribble-guided image editing has already been explored in prior work, particularly DreamOmni3 [1], which also proposes a synthetic data generation pipeline for multiple scribble-guided editing tasks. Furthermore, training with region-aware losses that emphasize edited regions is not new. While mosaicking has been explored in other tasks, such as personalization, I find its application to this setting interesting. Having said that, I still believe this work offers value.

[1] DreamOmni3: Scribble-based Editing and Generation

Questions for authors

Please refer to the 2nd point mentioned in the Weaknesses. I am generally leaning toward acceptance and only expect the authors to address this point during the rebuttal. The other points, such as incorporating additional curated data or providing further ablations of progressive training, are beyond the scope of the rebuttal and are not expected to be addressed. I intend to maintain my acceptance score if the second point is adequately addressed.

4 comments
AuthorsOpenReview ↗

Title: Response to Reviewer a8PY (1/2)

W1: Curated natural multi-task samples

Thank you for this suggestion. We agree that mosaicked samples differ from natural multi-edit scenes, and that a small curated set of multi-turn editing examples could help narrow this gap. Our results on VIBE’s natural multi-task examples (Table 3) show that Multi-Task Mosaicking already improves performance beyond the mosaicked training setting. We view the suggested data as complementary supervision for performing multiple edits within a shared scene. In future work, we will explore augmenting our training data with such examples, manually verifying that all requested edits are correctly executed and that subsequent edits preserve the results of earlier ones.

W2: Construction of Translation data

Thank you for the careful reading. Translation (TR) is explicitly included in both Layered Synthetic Data and Visually Coherent Synthetic Data, and is trained alongside Addition, Removal, and Replacement in Stage I and Stage II, respectively. The manuscript did not describe its construction clearly enough. We provide the details below and will incorporate them into the revised manuscript and release the corresponding data construction code.

1. Construction of source–target image pairs

Layered Synthetic Data. We sample a target layer ℓ and a destination on the canvas, move ℓ to that location, and re-composite the layer stack to obtain the target image Ĩ. The original location reveals the existing underlying layers, so no inpainting is required. Pixels outside the original and destination regions remain unchanged.

Visually Coherent Synthetic Data. A VLM proposes a semantically and spatially plausible translation, such as “move the air conditioner from the upper-left corner to the wall on the right.” An image editing model then generates candidate target images, and human curation selects visually coherent results.

2. Scribble synthesis and instruction generation

We obtain the object masks at the original and destination locations. For layered data, these are derived from the target layer’s alpha channel before and after translation. For visually coherent data, a referring segmentation model localizes the object in the source and target images.

Using these masks, we draw a referring scribble, such as a circle or box, around the source object and an arrow indicating its destination. An LLM then generates the corresponding instruction from the operation type, scribble type, and color—for example, “move the object circled in red to the location indicated by the arrow.”

3. Training

Translation pairs are included in both training stages together with the other three operations. In Stage I, layered translation pairs also participate in Multi-Task Mosaicking. Their edit-region weighting masks are computed from the source–target image differences, capturing changes at both the original and destination locations. This provides supervision for both removing the object from its original position and placing it at the requested destination.

W3: Comparison with mixed training

Thank you for suggesting this comparison. We have added mixed-training and reversed-order ablations to evaluate the benefit of our progressive schedule. All configurations use the same datasets, a total of 4,500 training steps, and a global batch size of 64.

  • Mixed single-stage training: We jointly train on Layered and Visually Coherent Synthetic Data. To account for their different sizes (180K vs. 4.5K), we follow the reviewer’s suggestion and vary the sampling ratio: Layered : Visually Coherent = 40:1, 10:1, 5:1, and 1:1.
  • Reversed-order training: We reverse the two stages while retaining their respective training steps: 500 steps on Visually Coherent Synthetic Data, followed by 4,000 steps on Layered Synthetic Data with Multi-Task Mosaicking.
TrainingIA (1-Task)VC (1-Task)
Mixed (40:1)89.6491.11
Mixed (10:1)89.1190.97
Mixed (5:1)88.1190.67
Mixed (1:1)90.4791.78
Reversed order88.6788.47
Ours (Stage I → Stage II)90.1693.81

Across the tested sampling ratios, mixed training achieves comparable IA but consistently lower VC. Our progressive schedule improves VC by 2.03 points over the strongest mixed baseline (1:1), while maintaining similar IA (90.16 vs. 90.47). Reversing the stage order reduces VC to 88.47, 5.34 points below our schedule. These results support the benefit of progressive training over the tested mixed and reversed-order alternatives under the same step budget, with the clearest improvement in visual coherence.

AuthorsOpenReview ↗

Title: Response to Reviewer a8PY (2/2)

W4: Methodological novelty

Thank you for the comment. We agree that synthetic data construction for scribble-guided editing has been explored in prior work, including DreamOmni3, and that region-aware losses are established techniques. Our contribution is a low-cost, scalable supervision construction and training scheme tailored to precise localization, visual coherence, and multi-task binding. Its design is motivated and validated by the following comparisons:

  • Supervision for precise and visually coherent editing. Our comparisons show that scalable layered supervision already supports strong instruction adherence, while visually coherent supervision improves visual integration (Sec. 3.1). These findings motivate a progressive schedule: first learn precise editing from re-renderable layers, then refine visual coherence using a small curated dataset. The additional mixed-training and reversed-order ablations further support the benefit of this schedule over the tested alternatives.
  • Supervision for multi-task binding. Our distractor-scribble control distinguishes exposure to multiple marks from supervision for executing multiple requested edits: adding distractor scribbles yields substantially smaller gains than providing complete multi-task targets (Sec. 3.2). Multi-Task Mosaicking constructs these targets by combining existing single-task image pairs and their corresponding scribble–instruction relationships, without requiring additional multi-turn image generation. Its effectiveness on VIBE’s natural multi-task examples demonstrates transfer beyond the mosaicked training setting (Table 3).
  • Edit-region weighting is an auxiliary component that uses the precise masks available from layered data without requiring additional mask annotation; we do not claim it as a novel loss.

Together, these results support the value of the proposed supervision scheme. On VIBE’s Deictic Level, it achieves the highest single-task score among the evaluated models and multi-task performance comparable to leading closed-source models.

I thank the authors for providing further clarification and experiments. I think the new ablation on the mixed-data training and its lower performance compared to the proposed progressive training add value to the contribution. Furthermore, the paper would be clearer if you add complete details of the data construction so readers can follow it easily. I maintain my original score of Weak Acceptance.

Thank you for your positive assessment and support. In the revision, we will include the additional experiments and provide complete data construction details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.