acceptodds
Withdrawn from ICLR 2027

VEGA: Sparse Mark Experts for Composable Visual-Instruction Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Precise image editing has long sought spatial instructions that express not only what to change, but also where to change. Modern multimodal editors are strong at understanding semantic intent from language, yet precise spatial grounding remains difficult to express through text alone. Visual instructions, such as boxes, arrows, and doodles, provide this missing grounding directly in image coordinates, while existing editors struggle to execute such instructions reliably. We study this failure and find that visual marks are not simply inaccessible to the editor. A frozen editor can localize and manipulate these marks when they are treated as image content, yet often fails when the same marks serve as editing instructions. This gap suggests that existing editors have not learned to reliably use visual marks as editing controls. We propose VEGA, a lightweight adapter that turns visual instructions into executable editing controls. For each type of visual instruction, VEGA trains a specialized expert using flow matching with auxiliary attention supervision, encouraging the expert to route the corresponding spatial evidence into localized editing responses. Compound training further encourages each expert to remain selective in the presence of competing annotations. We then introduce Attention Response Routing, which uses these learned responses as token-wise responsibilities to compose independently trained experts along a shared generation trajectory, without parameter merging or joint expert training. Experiments on internal and external benchmarks show that VEGA is effective across multimodal editing models, substantially improving spatially grounded editing and enabling efficient composition of multiple visual instructions while preserving the original generation quality.

Primary area:
generative models
Keywords:
Visual Control, Image Editing, Sparse Experts, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pZgROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes VEGA, a lightweight framework for visual-instruction image editing that directly executes in-image annotations such as boxes, arrows, text, and doodles. It trains instruction-specific LoRA experts with attention-routing supervision and selective execution, and introduces Attention Response Routing (ARR) to efficiently compose independently trained experts within a shared denoising trajectory. The authors further construct All-in-Image-45K and All-in-Image-Bench, and experiments on their benchmark and VP-Bench demonstrate improved spatial localization and visual-instruction execution over existing editors.

Critical strengths
  1. The paper identifies the “Reading without Routing” failure mode, offering a clear motivation for improving how pretrained editors translate visual annotations into localized editing actions.
  2. Attention-routing supervision is simple yet effective, with controlled ablations showing substantial improvements in execution accuracy and spatial localization.
  3. VEGA provides a lightweight and modular expert framework, enabling independently trained LoRA experts to be composed through ARR without joint expert training or inference-time edit masks.
Critical weaknesses
  1. ARR underperforms the simpler independent prediction mixing baseline on the core compositional metric (49.8% vs. 57.7% strict success), despite compositional editing being a central contribution of the paper. Although ARR improves Edit-IoU and efficiency, this result raises concerns about whether the proposed routing mechanism can reliably preserve and combine individual expert capabilities.
  2. The performance is highly uneven across instruction types. In particular, the Force/Arrow expert achieves only 29.7% accuracy individually and drops to 21.0% under composition, substantially below the other experts. Since directional object relocation is a fundamental visual editing operation, this weakness limits the claimed generality and composability of VEGA.
  3. I am concerned about the reliability of the evaluation protocol. Strict Execution relies on binary judgments from Qwen3-VL-4B, but the paper provides no human agreement study or other validation demonstrating that this relatively small VLM is a reliable judge for fine-grained spatial editing. Similarly, Outside PSNR/LPIPS is computed after an empirically fixed 16-pixel dilation without sensitivity analysis, making it unclear how robust the reported preservation advantages are to this design choice.
  4. Could the authors report results on VIBE [1]? Since VIBE directly targets visual-instruction-driven image editing and closely aligns with the core problem studied in this paper, it would serve as a particularly relevant external benchmark for VEGA.

[1] How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing

Questions for authors

Please refer to the weaknesses.

Reviewer_kHZ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VEGA, a framework for image editing through visual instructions drawn directly on the input image. It trains independent condition-side K/V LoRA experts using flow matching, auxiliary attention supervision, and selective execution training. At inference, Attention Response Routing (ARR) composes these experts through joint attention within a shared denoising trajectory. The paper also introduces All-in-Image-45K and All-in-Image-Bench and evaluates generalization to human-drawn instructions and an external benchmark.

Critical strengths

Clear and useful motivation. The distinction between perceiving visual marks and executing their intended edits motivates an interesting problem. Supporting boxes, arrows, text, and doodles through the existing visual-conditioning pathway offers a practical interface for spatially grounded editing.

Informative controlled ablations. Table 2 holds initialization, training data, and optimization settings fixed when evaluating attention supervision. The improvements in edit success from 63.3% to 73.0% and Edit-IoU from 39.5% to 66.8% provide meaningful evidence for this component. The selective-execution study further supports training experts to preserve unrelated instructions.

Strong single-control results and useful evaluation breadth. VEGA achieves 84.35% strict execution on the single-control benchmark, compared with 69.26% for the strongest comparison editor. Evaluation on VP-Bench, human-drawn annotations, repeated-object scenes, and natural scene text provides complementary evidence beyond the internally constructed benchmark. The dataset and planned release of code and checkpoints could also benefit subsequent research.

Critical weaknesses

The benefit of separate experts needs stronger isolation. The controlled ablation establishes the value of attention supervision, but the comparison with a single adapter trained across instruction families is less clear. The “universal soft” baseline in Table 7 is not sufficiently specified to determine whether it provides this control. A clearly defined joint-adapter baseline, with training data, updates, and parameter budgets reported, would better establish when independent specialization is beneficial.

Overlapping-edit inference is underspecified. Appendix E.2 introduces two initial steps using independent expert predictions and regional selection before ARR. However, the procedure for detecting overlap and selecting the stronger prediction within each region is not precise enough to reproduce. This also complicates the efficiency comparison: Table 4 reports measured relative generation time for ARR but forward-pass count for independent prediction mixing. Comparable wall-clock measurements, including initialization, would make the claimed advantage easier to assess.

Evaluation needs additional calibration and detail. Strict execution relies on automated judgments, while Edit-IoU depends on thresholded pixel differences. The exact completion prompts, change threshold, and image-alignment procedure are not fully specified. The human preference study provides useful complementary evidence, but one selection per example does not directly validate the execution judgments. Agreement with human completion labels on a small representative subset would strengthen confidence in the reported improvements.

Questions for authors

What proportion of composition examples uses the overlapping-edit initialization? How are overlapping regions identified without inference-time edit masks, and how much does initialization affect completion and runtime?

Does the universal-soft baseline represent a jointly trained adapter or a different composition rule? Please clarify its training and inference settings.

Can the authors report human agreement with strict-execution judgments and provide the exact Edit-IoU threshold and alignment protocol?

Reviewer_7FowOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VEGA, a framework for image editing using visual instructions such as text, bounding boxes, arrows, and doodles directly overlaid on input images. The method trains LoRA experts for different visual instructions and introduces an attention-based routing mechanism to combine them for compositional editing.

Critical strengths
  • The problem of improving controllability in image editing models is important and practically relevant.
  • The method demonstrates some improvements in spatial localization and instruction-following accuracy across visual editing tasks.
Critical weaknesses
  • The idea of providing visual cues directly on input images to guide editing is not new. Similarly, training additional lightweight adapters on top of pretrained instruction-based image editing models to incorporate visual cues has been explored before. The attention-based mechanism for combining these adapters is also not particularly novel. Overall, I find the methodological novelty rather limited, as the proposed framework largely builds on existing techniques.

  • The paper is poorly written. The transitions between sections and ideas are often abrupt, and concepts and methodological choices are not explained clearly or intuitively. Overall, the presentation lacks a smooth narrative, making it challenging to follow and understand the contributions.

  • I am not convinced of the benefits of encoding editing instructions directly into the input image rather than using a separate conditioning mechanism. In particular, it is unclear what advantage this approach offers over a lightweight adapter that processes the same visual instructions separately and injects the corresponding conditioning signals into a pretrained editing model. This concern is especially relevant for text-based instructions, where rendering text directly onto the image seems unnecessarily restrictive compared to providing it through the model's existing text-conditioning pathway or associating it with additional visual cues learned through lightweight adapters. I think the paper would benefit from a clearer justification of this design choice and experimental comparisons demonstrating its advantages over such alternatives.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.