- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how object-related visual tokens exert influence inside the language-model backbone of vision-language models. The authors operationalize object-token influence along three axes: erasure sensitivity, which measures how replacing object-token states affects the correct output; cross-instance transfer, which tests whether an object-token hidden state can causally transfer category-relevant information to another image; and category readout, which measures how well object category information can be recovered from the hidden representation. To compare these heterogeneous signals across depth, the paper converts each into a normalized layerwise mass profile and summarizes its location using a common half-mass depth statistic. Across seven adapter-style VLMs, the authors report a consistent depth ordering: erasure sensitivity concentrates earliest, cross-instance transfer later, and category readout deepest, yielding an intervention–readout gap. The paper further examines whether the effects of multiple object tokens combine additively, and finds that whole-set object-token interventions exhibit supra-additive effects relative to a matched background control, with this excess concentrated mainly in shallow layers and becoming indistinguishable from zero at deeper layers. Overall, the paper presents a systematic characterization of the depth organization and set-level interaction of object-token effects in VLMs, with potential implications for model intervention and visual-token pruning.
Critical strengths
- Extensive experiments and careful robustness analysis. The paper evaluates seven VLMs from five model families and includes a substantial set of controls and sensitivity analyses, including alternative baselines, prompt variations, threshold sweeps, pair-level statistics, and an LVIS extension. These analyses make the reported depth-ordering results reasonably well supported within the studied setting.
- A clear and unified framework for comparing different notions of object-token influence. The paper clearly distinguishes erasure sensitivity, cross-instance transfer, and category readout, and places these heterogeneous measurements on a common relative-depth coordinate using the same half-mass statistic. This provides a simple and interpretable way to compare where different forms of object-token influence concentrate across models.
Critical weaknesses
1. Limited scientific significance beyond descriptive characterization The paper carefully defines three operational notions of object-token influence and establishes a consistent depth ordering among them. However, the scientific story does not seem fully closed: it remains unclear what concrete model behavior this ordering explains, predicts, or enables. The paper does not connect the proposed influence profiles to downstream phenomena such as grounding, hallucination, robustness, or reasoning, nor does it show that the identified depths lead to better editing, intervention, or token pruning. The proposed applications to editing and pruning therefore remain largely speculative. Consequently, the work provides a strong answer to where these measurements concentrate, but a much weaker answer to why this organization matters or what can be done with it.
2. The category-readout axis does not establish object-token specificity The readout metric measures whether an object-token hidden state is more similar to text representations of its own category than to representations of other categories. This demonstrates category decodability from the selected token state, but does not establish that such information is specifically localized to object tokens. After contextual mixing, background tokens from the same image may also encode substantial information about the target object. A same-image background-token control—for example, comparing the similarity of object versus background tokens to the same category text—would help determine whether the reported readout is genuinely object-token-specific rather than reflecting more global image-level category information. Since the central result interprets as an axis of object-token influence, this distinction directly affects the interpretation of the main claim.
Questions for authors
-
What concrete model behavior is the proposed depth organization expected to explain or predict?
The paper establishes a consistent ordering, but it is not clear what observable VLM behavior this organization is intended to explain. Could the authors clarify the underlying scientific hypothesis behind these three notions of influence, and ideally test whether the resulting depth statistics correlate with or predict a concrete behavior such as object grounding, hallucination, robustness, or intervention effectiveness? Such evidence would help establish that the reported ordering is more than a descriptive regularity of the chosen measurements. -
Does the category-readout result remain object-token-specific when compared against same-image background tokens?
The current metric contrasts the similarity of an object-token state to same-category versus other-category text representations. However, after contextual mixing, background tokens from the same image may also encode information about the target object. Could the authors add a control comparing object and background tokens with respect to the same target-category text, e.g., or a corresponding double contrast against negative-category text? More importantly, does the reported , and hence the central ordering, remain qualitatively unchanged under such an object-vs-background-controlled readout? -
Can the authors demonstrate that the observed set-level non-additivity has a practical consequence for token selection or pruning? The paper motivates the non-additivity analysis partly through visual-token pruning, but currently only shows that the full-set intervention effect can exceed the sum of the two half-set effects. Could the authors test whether a tokenwise importance criterion actually makes worse pruning decisions at layers where is large, compared with a group-aware criterion that accounts for joint effects? This would substantially clarify whether the non-additivity result has actionable significance rather than being only an interesting intervention phenomenon.
-
How general is the full three-axis ordering beyond the specific COCO object-centric setting?
The LVIS extension provides useful evidence for , but it does not re-evaluate or the set-level non-additivity result. Could the authors test the full ordering, at least on a representative subset, under another dataset or a broader class of prompts/tasks? This seems important because the strongest conclusions currently rely on COCO segmentation-defined object tokens together with object naming and existence prompts, and it is therefore unclear whether the reported organization reflects a general property of VLM object processing or one specific to this experimental regime.
4 comments
Thank you for the detailed and thoughtful rebuttal. The additional experiments have addressed most of my concerns, and I intend to revise my rating upward.
I have one remaining question regarding the independence of the new validation. Since the cross-instance transfer peak is identified through activation patching, and the new answer-flip experiment also relies on token transplantation, could the authors clarify how this experiment provides independent evidence of practical significance beyond the original transfer metric?
Thank you for the follow-up, and for reconsidering the rating.
You are right that both use transplantation, so the flip experiment shares its intervention family with the transfer axis. It is not circular, however, and its main conclusion does not depend on the transfer metric.
1. Out-of-sample. For each model we drew 300 pairs and split them in half. The transplant layers were chosen on the first half from the cached depth profiles, by a rule fixed before the run, and the flip rate was measured only on the second half. The flip outcome played no role in choosing the layers. The experiment therefore asks whether a layer fixed in advance predicts where an edit works on new images.
2. Different unit and outcome. The transfer metric is a continuous change in the naming margin under single-token transplants. The flip experiment transplants the whole object-token set and records whether the model's top-1 answer actually becomes the donor category. A margin shift does not imply an answer change, so this is a behavioral outcome that the transfer metric does not guarantee.
3. The readout side is independent of patching. The readout axis is a similarity between hidden states and category text, with no intervention. The practical question is whether the layers where the category is most decodable are layers where an edit works. As noted in our response, the flip rate at the readout-peak layer is close to zero: at most 0.007 in six of seven models. In InternVL3.5-8B the readout profile has two peaks of nearly equal height, at and (within 0.034), and the pre-specified argmax rule selected the shallower one, where the flip rate is 0.17. At the deeper peak the flip rate is 0.000, as in the other models. This result does not use the transfer metric: decodability is not a reliable guide for choosing where to intervene.
We will describe the experiment in §4.3 as an out-of-sample test of intervention effectiveness and state that it shares the intervention family with the transfer axis.
Thank you for the detailed and thoughtful clarification. I find the explanation convincing. I appreciate the authors' efforts and have decided to raise my rating.
Thank you for the detailed review. We ran the controls you asked for. Results are below and will be added to the paper.
W2 / Q2. Object-token specificity of R
We computed the double contrast you suggest, Rg,i′(ℓ)=Rg,i(ℓ)−Rgbg(ℓ), where Rbg is Eq. (4) evaluated on same-image background states (patches no annotation touches). It contrasts object vs. background tokens and target vs. negative-category text at once.
So L50R, and with it the ordering, is unchanged under the controlled readout. With the paper's θ-based background set, the ordering also holds in 7/7. We use the double contrast because the single contrast sim(hobj,ty)−sim(hbg,ty) also picks up a category-independent offset between object and background states. This offset is largest in the two Qwen models, where it alone changes the order (post hoc diagnosis). The negative-category term removes it. We will report both.
W1 / Q1. What behavior the ordering predicts
The hypothesis is that the depth at which category information is readable from object tokens is not the depth at which it drives the output. This predicts intervention effectiveness, which we tested. We transplanted the whole object-token set from a donor image at each model's transfer-peak layer and at its readout-peak layer, and measured how often the answer flips on held-out pairs.
Layers selected by decodability are therefore layers where editing object tokens does not change the answer. For editing or steering, the profiles indicate which layers to use. We will add this experiment to §4.3.
Q3. Practical consequence for pruning
We ran the test on LLaVA-1.5-7B and Qwen2.5-VL-7B (200 pairs each) and added actual pruning on four models, using the attention-based rule of FastV (Chen et al., 2024) at a shallow layer near its default (ρℓ∈[0.08,0.11]).
The consequence is therefore for the pruning budget, not for the ranking. A tokenwise criterion picks the right tokens but underestimates the loss as more of an object is removed. We will add these results to §5.3.
Q4. Beyond COCO
We measured R on the LVIS pool of Appendix F, so the C–R ordering is now also evaluated there. R needs positive phrases in the text bank, which 247 of the 385 donor categories have (1,214 pairs).
The set-level analysis remains on COCO. Its robustness is supported by the actual-pruning result in Q3 and by two checks on COCO: it holds in the lowest third of clean margins (7/7) and under re-drawn random half splits (two models).
W1 (significance), in short. The ordering is not only a description of where measurements concentrate. It separates layers where object tokens can be acted on from layers where they can only be read (Q1), and the set-level result changes how much pruning a tokenwise score can justify (Q3).