acceptodds
Under review as a conference paper at ICLR 2027

Context-Aware Saliency Control for Diffusion-Based Object Modification

Abstract

Modern diffusion-based inpainting models and image-conditioned adapters provide reference-guided object insertion, but offer limited explicit control over the prominence of generated content. We introduce CASAdapter (Context-Aware Saliency Adapter), a module with 9.7M trainable parameters that modulates an object's CLIP image embedding according to the surrounding scene. Its orthogonal tangent-space update help to preserves the embedding norm after renormalization and bounds its angular displacement. This geometric constraint motivates limiting conditioning drift, without guaranteeing preservation of the generated object's identity. CASAdapter is trained through truncated DDIM sampling using a UniSal-based objective together with semantic and color-gradient regularization. Evaluation on a 300-image MS-COCO validation subset shows positive mean saliency gains under UniSal, DeepGaze IIE, SUM, and both TranSalNet variants at positive modulation settings, and negative mean gains at the tested negative setting. These observations support transfer beyond the training predictor. Larger orthogonal modulation produces higher mean saliency gains alongside lower CLIP similarity. Compared with the reported affine setting, the orthogonal variant offers a more favorable observed trade-off for four of the five predictors at a moderate setting; the strongest tested setting exceeds the affine setting in mean gain under all five predictors while retaining higher CLIP similarity.

Primary area:
generative models
Keywords:
Diffusion Models, Image Inpainting, Visual Saliency, Context-Aware Conditioning, Object-Level Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DnJMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CASAdapter for controlling the visual saliency of a selected object. It modifies the object’s CLIP image embedding based on scene context while keeping CLIP, IP-Adapter, and the diffusion inpainting model frozen. The adapter predicts a gated update orthogonal to the original embedding, then normalizes and rescales the result to preserve its norm and bound the angular change. Training combines saliency, semantic consistency, and color-gradient losses. The method is evaluated on 300 images from the COCO validation set. The authors report predicted saliency changes and CLIP similarity at different modulation strengths. Positive modulation increases the object’s average share of predicted saliency across several saliency models, while stronger modulation reduces similarity to the reference object.

Critical strengths
  1. CASAdapter works with a frozen reference-guided diffusion pipeline and requires training only a small modulation module. Its signed control parameter allows users to increase or decrease the target object’s predicted saliency.

  2. The evaluation includes several saliency models beyond UniSal, which provides the training signal. The paper also acknowledges that these predictions do not establish control over human attention or guarantee preservation of object identity.

Critical weaknesses
  1. Object-level saliency editing has been studied before, but the paper does not adequately discuss closely related work or establish a clear advance over existing methods. The proposed module combines scene cross-attention, gating, and an orthogonal update followed by normalization, without ablations showing the separate contributions of scene context and orthogonal projection. Preserving the embedding norm and bounding its angular change do not, by themselves, ensure better preservation of the generated object’s semantics or appearance.

  2. General-purpose image editing models already support local changes to color, lighting, and appearance. The paper neither compares against these models nor includes simple brightness, saturation, or contrast adjustments as baselines. The results therefore do not establish why saliency enhancement requires a dedicated adapter, or whether the proposed method offers advantages in control consistency, object fidelity, or computational cost.

Questions for authors
  1. How were the 300 test images selected? When an image contains multiple instances, how was the target object chosen? Were any filters applied based on object category, size, or initial saliency?

  2. Could you provide ablations without scene context and with a non-orthogonal residual update that retains norm normalization? These would help distinguish the contributions of context, normalization, and orthogonal projection.

  3. What is the exact formulation of the affine baseline, and why is only a modulation strength of 0.15 reported? Could you evaluate several strengths and compare the two methods at matched saliency gains or reference similarity?

  4. How does CASAdapter compare with simple brightness, saturation, and contrast adjustments, as well as representative general-purpose instruction-based editing models? Do these comparisons show an advantage in the consistency of saliency control or object fidelity?

Reviewer_3utROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CASAdapter learns a 9.7M-parameter module upstream of a frozen CLIP/IP-Adapter diffusion inpainting pipeline. Object-conditioned cross-attention aggregates masked scene features; a gated orthogonal update preserves the reference embedding norm and bounds its angular displacement by arctan(|γ|). Training uses UniSal saliency, CLIP similarity, and HSV-gradient losses, with gradients retained through the final two DDIM steps. On 300 COCO validation images, with four generations per image and matched seeds, positive settings increase mean predicted saliency share under five predictor configurations and a negative setting decreases it. Table 1 reports a saliency–CLIP trade-off and compares unmodulated, prompt, affine, and orthogonal settings. These are predictor-level effects, not demonstrated human-gaze or exact-identity control.

Critical strengths

S1:A concrete, economical conditioning interface with correctly scoped geometry. Equations 1–3 genuinely preserve the input norm and provide an angular budget, while the paper explicitly avoids claiming that this guarantees generated-image identity. This makes the mechanism understandable and technically defensible. S2:The evaluation includes meaningful controls and transfer beyond the training predictor. At γ=0.3, predicted relative share gains are 8.13–18.72% across all five configurations with CLIP 0.821, versus 0.838 for unmodulated inpainting; γ=-0.3 gives negative means across all five. Matched images/seeds, a saliency-oriented prompt baseline, and an affine variant make this more informative than a single-proxy demonstration.

Critical weaknesses

W1:Related-work positioning and attribution of the proposed design are incomplete. Look here! (Mejjati et al., ECCV 2020) already takes an image and mask, learns attention-redirecting foreground/background edits on curated MS-COCO data, supports attention amplification/attenuation and adjustable strength, and reports user studies. Deep Saliency Prior (Aberman et al., CVPR 2022) and Realistic Saliency Guided Image Enhancement (Miangoleh et al., CVPR 2023) further establish saliency-guided local editing, including generative operators and realism-aware enhancement. These works do not establish equivalence to a frozen IP-Adapter conditioner, but their omission leaves the incremental problem/method contribution inadequately positioned. The introduction’s “comparatively little attention” characterization needs qualification, not a blanket claim that the inverse problem is unstudied. Table 1’s single affine point and missing no-context control also leave the value of context and orthogonal modulation partly unisolated; the SUM gain at orthogonal 0.5 is below affine 0.15 (39.28% versus 51.21%). A task-aligned discussion and a small available prior/simple-edit baseline comparison are needed to identify the new capability. W2:The reliability of controllability is not established at the image level. The paper reports mean relative gains, which can be dominated by small original saliency masses: SUM at γ=0.8 is 73.97±192.79%, while the change in mean absolute share is only 4.854 percentage points (Sec. 5.4). There are no median effects, per-image success rates, paired intervals, or monotonicity proportions. Together with training-coupled CLIP and old qualitative settings in Fig. 3, the evidence cannot yet establish how often useful control succeeds while preserving recognizable appearance. This limits the practical significance of the central contribution.

Questions for authors

Q1:On stored outputs for the 300 images, can you report paired absolute saliency-share changes, medians, intended-sign success rates and image-clustered confidence intervals for the four γ settings? How often is the positive-γ response monotonic within an image? Q2:Can you compare a small affine-strength sweep and a no-context ablation on the same images at comparable CLIP similarity, rather than equating coefficient values? Please include representative current-benchmark outputs and failures to assess visual usefulness. Q3:Please specify training data size/duration and γ distribution, exact inpainting/IP-Adapter checkpoints, crop/compositing rules and the seed protocol for four generations per image. Does evaluation cover only original-scene reference crops or also external-reference insertion? Please include available hardware-specific cost. Q4:What capability does CASAdapter add beyond Look here! (ECCV 2020), Deep Saliency Prior (CVPR 2022) and Realistic Saliency Guided Image Enhancement (CVPR 2023)? On a small existing-image subset, can one compatible prior method or local contrast/saturation baseline be compared at matched saliency gain and comparable fidelity, with editable-region differences disclosed? Q5:How were the 300 COCO images and targets selected, and what are their category, object-size and initial-saliency distributions? Please specify the tested prompt and its selection rule; a few predeclared alternatives on a fixed subset could clarify whether the prompt baseline is representative. Q6:What evidence separates the semantic and HSV-gradient regularizers’ effects? Please report available ablations jointly on saliency and fidelity, and use existing outputs to assess color/saturation or boundary/background changes, including possible artifacts from fixed 10-pixel padding.

Reviewer_LZj1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an approach to control the saliency of the inserted object in the problem of inserting an object into a specified region of an original image to generate a target image. Its main contribution is a new method that achieves control over the saliency of the inserted object. Specifically, the contributions of this paper include the following aspects: first, it proposes to perform saliency control during the object insertion process; second, it designs a learnable saliency modulation network, CASAdapter, which is inserted between the CLIP image encoder and the diffusion inpainting model; third, it designs a saliency loss function that constrains the saliency of the inserted region in the generated image during training. CASAdapter is a lightweight learnable network, and all other borrowed modules are frozen during training, so training is highly efficient.

Critical strengths
  1. Proposes a new type of content generation control, namely saliency control over the region where a new object is inserted in the image.
  2. Provides a concrete saliency control method.
Critical weaknesses
  1. The innovation of the proposed method is incremental, because adding a learnable adapter to a generative model to achieve some kind of control over the generated image while not significantly increasing the training burden is a common approach in inpainting.
  2. The proposed saliency control method comes at the cost of the semantics of the inserted object; when controlling saliency, the inserted content changes to varying degrees.
  3. The authors' control over the direction of change does not seem very intuitive.
  4. Several loss functions are not given explicit formulas, which makes it inconvenient for readers to understand them accurately. For example, Lsal\mathcal{L_{sal}} is described by the authors as "The loss is the ratio between the mean predicted saliency inside the target mask in the original image and that in the modulated output."
Questions for authors
  1. Although the authors provide a parameter gamma for controlling saliency strength, the relationship between the value of this parameter and the actual saliency of the inserted object in the target image is not very clear, and the control is not intuitive.
  2. Why the feature vector needs to be updated in the tangent space is not clearly motivated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.