- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how replacing visual activations can make a frozen VLM report an object absent from the image, without changing the image or model weights. Through statistical controls, the authors motivate a Gaussian concept bank defined by class means and low-rank covariances, and evaluate insertion across multiple models, COCO, and LVIS.
Critical strengths
I am glad to see the paper ask why removing and inserting objects behave differently, rather than simply introduce another activation-editing method. Holding coverage fixed while varying activation diversity, and comparing mean, marginal-preserving, and Gaussian surrogates, makes the question concrete and testable. The differences among these conditions in Table 1 provide useful evidence that the structure of the substituted activation set matters.
Critical weaknesses
I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.
-
My most immediate concern is Table 6 in Appendix B. It states that comparisons use the same 150 images and are paired against the proposed method. An injection rate of 0.613 implies approximately 92 successes, while 0.320 for the text direction implies 48: the net difference should be 44. The reported discordant counts, 39:12, instead imply a difference of 27. Likewise, 0.007 for residual steering implies approximately one success, giving a difference of 91 rather than the reported 74:0. For the same paired binary outcomes, these quantities must agree. These may be reporting errors, but please identify the samples and reference run for each row and provide the underlying 2×2 counts before relying on these significance claims.
-
In my view, the experiments strongly support the claim that target-related covariance improves insertion, but not the stronger conclusion that insertion requires target-specific second-order structure. The tuned text direction already achieves 0.320 injection, and the shared-covariance bank reaches 0.440 (Table 6; Section 4.4). Moreover, independent feature-wise permutation disrupts higher-order cross-dimensional dependencies as well as covariance. A successful Gaussian surrogate shows that second-order statistics provide an effective construction, not that they are the only explanation. I would narrow the necessity claim or add non-Gaussian surrogates with the same mean and covariance but different higher-order structure.
-
Section 3 explains the surrogate statistics clearly, but every substitute is subsequently rescaled to the norm of the token it replaces. This token-dependent nonlinear operation changes the moments of the activations actually written into the model. For example, tiling one vector and then applying unequal norm scales generally creates variation along that direction, so zero covariance strictly describes the set before rescaling. The authors acknowledge moment perturbation, but do not quantify it. Because the mechanistic argument depends on these controls, I would like measured mean and covariance errors before and after normalization, together with a common-norm or alternative-normalization control.
-
Higher insertion rates are meaningful, but faithful local editing needs a separate assessment. In Table 6, the Gaussian has retention 0.376 versus 0.535 for the text direction. Appendix B imposes a retention floor of 0.35 rather than comparing methods at matched fidelity. The human study in Appendix I checks whether the target is described as present, not whether other objects, attributes, and relations remain correct. I would like an insertion–fidelity curve and a human or fact-level evaluation of untouched scene content. This does not invalidate the mechanism finding, but it matters for the proposed use as controlled counterfactual editing.
Minor Weaknesses:
-
Appendix G explicitly places LVIS bank estimation in the training split, but the relationship among COCO bank images, donor images, and evaluation images is less clear. Please provide the splits and image-level deduplication rule. I am not alleging leakage; I would simply like this separation to be directly checkable.
-
The paper already notes that a non-significant difference does not establish equality, which I appreciate. Please make that qualification more prominent and report confidence intervals for paired effects, especially for borderline results across multiple rank comparisons. Table captions should also identify independently resampled runs so that values from different tables are not mistaken for one shared realization.
Questions for authors
For clarity and to facilitate the rebuttal process, I have already outlined the key strengths and weaknesses that informed my rating in the sections above. Please refer to the Strengths and Weaknesses sections for the detailed rationale. I would be willing to reconsider my rating if the authors adequately address my main concerns during the rebuttal.