acceptodds
Under review as a conference paper at ICLR 2027

Concepts Are Distributions: Training-Free Insertion in Vision-Language Models

Abstract

Many activation-steering methods represent a concept as a direction in activation space. We test whether this representation also supports object insertion in the visual tokens of frozen vision-language models (VLMs). We find a clear asymmetry. A single direction can suppress an object, but it does not reliably introduce an object that is absent from the image. Learned directions, text-derived directions, and individual real activations all perform poorly. We show that the limitation is distributional. Repeating one concept vector across visual tokens can match the mean while collapsing covariance. A Gaussian whose population mean and covariance match those of real object activations produces responses comparable to transplanting a real object, whereas a surrogate that preserves the marginal statistics but destroys covariance does not. Existing feature-transfer operators follow the same ordering as the second-order structure they transfer. We therefore store each concept as a low-rank Gaussian estimated from a few images. On CLIP-L/14, four covariance directions are sufficient, giving a 20 KB representation per concept. Other vision encoders require modestly higher rank. The method requires no training and no reference image at inference time, transfers across five models and three vision encoders, and works without object-location annotations. These results show that object insertion depends on second-order visual-token structure and identify the visual-to-language activation stream as a security-sensitive interface when internal activations can be modified.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, activation steering, mechanistic interpretability, representation editing, concept insertion, multimodal models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_16C7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how replacing visual activations can make a frozen VLM report an object absent from the image, without changing the image or model weights. Through statistical controls, the authors motivate a Gaussian concept bank defined by class means and low-rank covariances, and evaluate insertion across multiple models, COCO, and LVIS.

Critical strengths

I am glad to see the paper ask why removing and inserting objects behave differently, rather than simply introduce another activation-editing method. Holding coverage fixed while varying activation diversity, and comparing mean, marginal-preserving, and Gaussian surrogates, makes the question concrete and testable. The differences among these conditions in Table 1 provide useful evidence that the structure of the substituted activation set matters.

Critical weaknesses

I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.

  1. My most immediate concern is Table 6 in Appendix B. It states that comparisons use the same 150 images and are paired against the proposed method. An injection rate of 0.613 implies approximately 92 successes, while 0.320 for the text direction implies 48: the net difference should be 44. The reported discordant counts, 39:12, instead imply a difference of 27. Likewise, 0.007 for residual steering implies approximately one success, giving a difference of 91 rather than the reported 74:0. For the same paired binary outcomes, these quantities must agree. These may be reporting errors, but please identify the samples and reference run for each row and provide the underlying 2×2 counts before relying on these significance claims.

  2. In my view, the experiments strongly support the claim that target-related covariance improves insertion, but not the stronger conclusion that insertion requires target-specific second-order structure. The tuned text direction already achieves 0.320 injection, and the shared-covariance bank reaches 0.440 (Table 6; Section 4.4). Moreover, independent feature-wise permutation disrupts higher-order cross-dimensional dependencies as well as covariance. A successful Gaussian surrogate shows that second-order statistics provide an effective construction, not that they are the only explanation. I would narrow the necessity claim or add non-Gaussian surrogates with the same mean and covariance but different higher-order structure.

  3. Section 3 explains the surrogate statistics clearly, but every substitute is subsequently rescaled to the norm of the token it replaces. This token-dependent nonlinear operation changes the moments of the activations actually written into the model. For example, tiling one vector and then applying unequal norm scales generally creates variation along that direction, so zero covariance strictly describes the set before rescaling. The authors acknowledge moment perturbation, but do not quantify it. Because the mechanistic argument depends on these controls, I would like measured mean and covariance errors before and after normalization, together with a common-norm or alternative-normalization control.

  4. Higher insertion rates are meaningful, but faithful local editing needs a separate assessment. In Table 6, the Gaussian has retention 0.376 versus 0.535 for the text direction. Appendix B imposes a retention floor of 0.35 rather than comparing methods at matched fidelity. The human study in Appendix I checks whether the target is described as present, not whether other objects, attributes, and relations remain correct. I would like an insertion–fidelity curve and a human or fact-level evaluation of untouched scene content. This does not invalidate the mechanism finding, but it matters for the proposed use as controlled counterfactual editing.

Minor Weaknesses:

  1. Appendix G explicitly places LVIS bank estimation in the training split, but the relationship among COCO bank images, donor images, and evaluation images is less clear. Please provide the splits and image-level deduplication rule. I am not alleging leakage; I would simply like this separation to be directly checkable.

  2. The paper already notes that a non-significant difference does not establish equality, which I appreciate. Please make that qualification more prominent and report confidence intervals for paired effects, especially for borderline results across multiple rank comparisons. Table captions should also identify independently resampled runs so that values from different tables are not mistaken for one shared realization.

Questions for authors

For clarity and to facilitate the rebuttal process, I have already outlined the key strengths and weaknesses that informed my rating in the sections above. Please refer to the Strengths and Weaknesses sections for the detailed rationale. I would be willing to reconsider my rating if the authors adequately address my main concerns during the rebuttal.

Reviewer_P72gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a frozen vision-language model can be made to report an object that is not in the image by editing only the visual tokens inside an object's box, without training. It finds that adding a single "concept direction" to those tokens rarely works, and argues that a real object's tokens vary, so a believable object needs both the right average and the right spread. It stores each concept as a mean plus four directions of variation estimated from up to twelve images, at 20 KB per concept, samples replacement tokens from it, and tests this on COCO and LVIS across five models and three vision encoders.

Critical strengths
  1. A clear, well-controlled finding: adding a concept direction names the object in only 13.8% of captions and the best direction from prior work reaches 32.0%, while sampling tokens with the object's mean and spread reaches 52.7%, not significantly different from pasting a real object's tokens at 50.0%.
  2. The method is simple, training-free and widely replicated: four directions suffice on CLIP, one stored bank works across three language models that share that encoder, and the effect repeats on LVIS and on two other encoders, which need eight directions.
Critical weaknesses
  1. The evaluation shows that the model names the object, not that it sees a coherent scene. The paper says the edit makes the model "report a giraffe and integrate it with the scene", but success is only whether the caption names the target, and the human check covers one method and asks only whether the caption describes the target as present. The one attribute test, asking the colour of the inserted object, could not separate the cases because the model also named a colour for objects it had denied seeing, so count, attributes and relations to the scene remain untested.
  2. The comparison has only one competitor built for this task and leaves out the simplest alternative. Most compared methods come from work on reducing hallucination, removing hallucinated objects and text-only language models, and the one method built for object replacement, VISTA-SAE, also edits along a single feature direction, so the comparison mainly repeats what the paper's own controls already show: one-direction edits do not insert objects. Pasting or inpainting the object into the image is never compared, although it works on any model, while the paper's method needs what the paper itself calls "a white-box attack on the inference pipeline.
  3. The introduction and conclusion overstate what the paper shows about concept-specific spread. The conclusion says insertion "requires target-specific second-order structure", but a spread pooled across all 73 concepts already gives 0.360, a shared 64-direction bank gives 0.440, which is not significantly different from pasting a real object, and the concept's own spread gives 0.600. Section 4.4 states it more carefully, that spread "first makes the mean readable and then contributes concept-specific information", and the headline claim should be worded the same way.
Questions for authors
  1. Can you report target naming and caption retention at every swept strength for every method, including VISTA-SAE, so that methods can be compared at equal retention rather than at single chosen points?
  2. Would blinded raters shown the image and the edited caption judge the inserted object's count, attributes and relation to the scene as consistent, which is needed to support the claim that the model integrates it with the scene?
  3. Does the pooled-spread control that gives 0.360 use the same four directions as the concept bank that gives 0.600, since otherwise the gap mixes concept-specific information with the number of directions?
  4. How does simply pasting or inpainting the object into the photo compare on the same images and metrics, since that needs no white-box access and would show what editing the visual tokens offers beyond editing the image?
Reviewer_58qeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper makes an interesting observation about VLM interventions: concpet removal is easier than insertion. They specifically focus on the interface between the vision encoder and the LLM in LLaVA-like VLMs. So at the visual tokens that come out of the vision encoder, they try to either introduce a new object into the image (e.g. giraffe) or remove an existing object. Intervention success is defined as how often the generated caption changes (or other variants such as yes/no questions). The main bulk of the paper focuses on analyzing how exactly insertion can work better: inserting the same copy of the avg. concept token does not work well but modeling the distribution of the concept tokens works better. For example, taking all the tokens that are on giraffe regions across various images, then storing their covariance matrix, and replacing them with "sampled" giraffe tokens.

Critical strengths

This asymmetry is a genuinely cool insight that I was not aware of and the authors did due justice empirically to show it: across several LLMs and encoders, two datasets, and many variants of inserting concept tokens.

Critical weaknesses

The biggest issue was that it was hard to follow the paper and often the main point or key intuition got lost:

  • The abstract overall reads very dense and complicated. E.g. the very last sentence, instead of providing a bigger picture outlook, reads is technical jargon-dense point: “These results show that object insertion depends on second-order visual-token structure and identify the visual-to-language activation stream as a security-sensitive interface when internal activations can be modified.” → e.g. what is a security-sensitive interface? Why is security suddenly related to this paper? After reading the abstract I did not get that sense at all. The introduction expands a bit on this, e.g. how the interface between vision encoder and LLM is a “natural point” for modifying the model without touching the weights or input image. I am not sure this is a good motivation… if you have access to “internals” of the VLM, ie activations between input and output, then you can also do a lot more than just tamper with the boundary between vision encoder and LLM usually, i.e. activations anywhere in encoder or LLM.
  • Similarly the introduction has weird notation such as “11:15” which is never introduced and seems to be only explained in Sec4? Reading feels just hard to follow. A lot of confusing sentence such as “Our mechanism predicts that a tiled mean should be insufficient for insertion.” → How can a mechanism predict anything? And I am not sure I would call this a mechanism but that’s perhaps okay.
  • The results/analysis section is also a bit hard to follow: 4.2 introduces analysis with similar notation/ideas as the main results (such as low-rank) but it is not quite the main method. It could help to more clearly present the main method first and then change it in later sections; especially since 3.2 and 3.3 had just introduced the main method.
  • Finally, some inconsistencies/overclaims such as “single rank-4 bank estimated under Vicuna-7B transfers to the other two models without refitting” → the features are independent of LLM so this is a trivial statement, no?
  • 4.6 feels very abrupt, no re-stating of what this subsection tries to answer, how this method differs to insertion, or what eg the variable P is

So my main issue is that the paper points out a simple cool insight but then spends so much time on specific details of the Gaussians and ranks with almost no discussion or intuition on the nature of this asymmetry.

The best single-direction insertion baseline from Jiang et al (0.32) is not far below single-direction removal (0.55), so “removal works, insertion doesn't” seems overstated in the intro with 13.8% from a weaker baseline.

(Some weird formatting issues with citations throughout paper such that they appear kind of randomly.)

Questions for authors

Could you spend more time in the paper actually characterizing and explaining this asymmetry you highlight as a contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.