acceptodds
Under review as a conference paper at ICLR 2027

Minimum-Sufficient Selective Control for Instruction-Based Image Editing

Abstract

Instruction-based image editing handles many simple requests well, but difficult edits still expose a control-allocation problem: weak intervention often under-edits hard cases, while uniformly strong intervention increases spillover and cost. We cast editing as minimum-sufficient control allocation over a locked bounded-cost intervention family with explicit end-to-end efficiency accounting. We keep this intervention family fixed and learn a lightweight controller from deployment-time observables. The policy first decides whether to leave the temporal anchor, then selects the promoted temporal horizon, and finally applies deterministic localization to the escalated branch. On the locked MagicBrush and PIE-Bench++ benchmarks, the full policy FO improves over the temporal-only anchor TO from 0.7607 to 0.8481 edit success and from 22.9583 to 24.7291 composite score while staying within the same efficiency envelope, with sub-second latency and low VRAM. Matched-budget controls, multi-seed confidence intervals, abstraction-transfer tests, and oracle diagnostics support that the gain comes from better allocation rather than a uniformly heavier editing path.

Primary area:
generative models
Keywords:
instruction-based image editing, diffusion models, selective control, adaptive routing, spatial refinement, image preservation, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NE4POpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the efficiency–quality tradeoff in instruction-based image editing by framing model execution as an adequacy-constrained, minimum-sufficient control allocation problem. Instead of uniformly deploying heavy structural conditioning or relying on an unconstrained weak editor, the authors propose a frozen intervention space comprising InstructPix2Pix and various temporal/spatial configurations of SD 1.5 with Canny ControlNet. A lightweight controller routes samples through a three-stage cascade: first predicting whether structural intervention is required, then choosing an appropriate ControlNet denoising horizon, and finally applying deterministic spatial refinement via saliency and directional instruction keywords. Evaluated on MagicBrush and PIE_Bench_pp, the proposed policy improves edit success and composite scores over fixed-budget anchors while operating under a sub-second latency budget.

Critical strengths

1. Principled Allocation Framing: Formalizing image editing as an adequacy-first, minimum-sufficient control allocation problem is conceptually clean and practically motivated. The lexicographic tie-breaking and adequacy thresholds are well-defined, and the matched-budget experiments convincingly show that gains come from targeted routing rather than arbitrary escalation.

2. Rigorous Ablation and Diagnostics: The paper features solid empirical rigor within its evaluated scope. The authors systematically isolate temporal horizon selection from spatial blending, provide failure slice analyses, report bootstrap confidence intervals across random seeds, and validate the allocation behavior against meaningful negative controls and human evaluation.

Critical weaknesses

1.Outdated Backbones and Limited Modern Relevance: The action space is restricted to InstructPix2Pix (2023) and SD 1.5 + Canny ControlNet (2023). The dilemmas investigated—such as background bleeding in the base editor and over-rigidity from edge conditioning—are largely artifacts of early U-Net diffusion models. Contemporary architectures (e.g., DiT-based models, modern flow-matching backbones, and instruction-tuned MLLM editors) have inherently resolved these localization and adherence challenges without external edge detectors. Tellingly, UltraEdit SD3 decisively outperforms the proposed pipeline across all editing metrics in Table 1, casting doubt on whether this complex routing framework remains relevant or necessary for modern editing stacks.

2.Fragile Spatial Heuristics and Feature Leakage: The spatial refinement module relies on an ad-hoc heuristic combining low-level image gradients with crude keyword matching for cardinal directions ("left", "top", etc.). This mechanism is brittle and fails on relational or non-spatial instructions. Furthermore, the 36-D controller feature vector leaks ground-truth benchmark metadata, including dataset IDs and mask-geometry features. This reliance on benchmark-specific cues undermines claims of real-world applicability, as evidenced by the severe drop in performance under cross-dataset transfer (Table 8).

Questions for authors

1.Portability to Modern Backbones: How does the proposed control framework translate to modern architectures (e.g., FLUX, SD3, or unified DiT editors) where external structural guidance like Canny edge conditioning is rarely needed?

2.Performance Without Benchmark Metadata: In the 21-D "deployment core" feature setting that strips dataset identity and mask geometry (Appendix Table 11), what are the complete Target, Preservation, Composite, and Success scores on both benchmarks?

3.Complex Relational Grounding: How does the deterministic Stage III spatial gate handle complex relational phrasing (e.g., "the cup between the two books") or non-cardinal spatial modifiers without introducing visible boundary artifacts?

Reviewer_6pmDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of control resource allocation in challenging image editing tasks. Insufficient intervention may result in incomplete target edits, while uniformly applying strong intervention to all samples may introduce unintended modifications to non-target regions and incur unnecessary inference costs. To address these challenges, the authors train a lightweight controller over a predefined set of editing operations. Using observable features extracted from the original image and editing instruction, the controller first determines whether a sample should be escalated from a low-cost editing path, then selects an appropriate control configuration for escalated samples, and finally spatially localizes the editing results. The goal is to allocate the minimum intervention necessary for each sample to successfully complete the target edit while preserving irrelevant content as much as possible. Extensive experiments demonstrate that the proposed method improves editing success rates and overall editing quality while reducing unintended modifications to non-target regions, with negligible additional inference cost.

Critical strengths

(1) The problem is practically meaningful. Different images and editing instructions require different levels of intervention. This paper formulates editing success, preservation of irrelevant content, and inference cost within a unified control allocation framework. By exploring sample-adaptive selection of editing paths and control configurations, it provides a valuable perspective on the trade-off between editing quality and computational cost.

(2) The experimental evaluation is relatively comprehensive. In addition to the main experiments, the paper presents analyses of challenging samples, comparisons of decision quality under different computational budgets, transfer and generalization studies, and human preference evaluations. These results provide further evidence supporting the central claim that allocating control resources on demand is effective.

(3) The improvements in quality and efficiency are practically meaningful. Compared with TO, FO achieves noticeable improvements in editing success rates and content preservation quality, while introducing only minimal changes in latency and memory consumption. Per-sample comparisons further indicate that challenging samples benefit substantially from the proposed method.

Critical weaknesses

(1) Limited methodological novelty. The proposed method is primarily a pipeline-based combination of existing editors, control levels, and spatial masks, making it overly engineering-oriented rather than methodologically innovative. Furthermore, the paper does not evaluate the proposed approach using more recent image editing models, such as those introduced in 2025 and 2026, making it difficult to assess its competitiveness and practical applicability.

(2) Lack of robustness analysis. The five control levels in TO involve several factors, including editor type, sampling steps, classifier-free guidance (CFG) scale, edge guidance strength, and the time interval during which guidance is applied. However, the paper lacks a systematic parameter sweep and sensitivity analysis of these key factors, making it difficult to assess the robustness of the proposed method to different control configurations.

(3) The ablation studies do not sufficiently establish the necessity and synergy of individual components. The existing FO–TO comparison provides useful evidence for the effectiveness of spatial refinement. However, the paper does not examine the effect of removing each of the five candidate operations individually. Nor does it separately isolate the contributions of the two routing decisions—whether to escalate and which control level to select after escalation—under the complete five-level setting and matched computational budgets. Consequently, it remains unclear whether both the five-level design and the two-stage routing mechanism are essential to the observed improvements.

Questions for authors

Please see the Weaknesses section.

Reviewer_P8VrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper treats instruction-based image editing as a routing problem. It does not build a new editor. Instead it fixes five actions to keep ControlNet on for longer and longer during denoising. It then trains a small controller to pick among them.

  • The controller works in three stages. Training labels come from running all five actions on a calibration set and taking the weakest one that passes the thresholds. A random forest on 36 hand-crafted features decides whether and how strongly to escalate. The escalated output is then blended with the source image through a fixed saliency mask, so everything outside the mask is copied from the original.

  • On MagicBrush and PIE-Bench++, 3 stages outperform 2 stages. The authors argue this shows the gain comes from smarter allocation, not from a heavier editor.

Critical strengths
  • I like that latency and VRAM are measured end to end for every method on the same GPU.
  • Efficiency is treated as a real metric here. And the analysis goes further than most editing papers.
  • The limitations section is honest about dataset indicators, asymmetric transfer and the heuristic mask.
Critical weaknesses
  • One of my main concerns is that the learned part adds very little. At a matched budget the gate beats random selection by only about 1 point. Stage II is never isolated, and there is no test of the blend applied without the gate, so the better allocation claim isn't really supported.

  • Another concern is that the gain mostly comes from pasting source pixels back. On MagicBrush both scores are PSNR, so copying the source alone pushes them up, and the baselines don't get the same blending. It would help to see the blend applied to the baselines, and results from a VLM-based judge or the official benchmark metrics.

  • Last, modern editors in the past two years handle easy and hard edits in one model, so the editors used here feel outdated. Since most of the gain comes from preserving the source outside the edit region, methods that do this more directly, such as KV-Edit, should be discussed and compared.

Questions for authors
  • Can you give a per-stage ablation on one consistent configuration: gate + fixed F0, then TO, then FO?
  • Is there evidence that a cheap-vs-strong gap still exists for modern editors, so that routing remains useful?
  • The routing idea seems closer to agent-style editing with verification and retry, or test-time scaling. How does your approach compare to these?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.