acceptodds
Under review as a conference paper at ICLR 2027

Object-Token Influence in Vision–Language Models: Where It Lives and How It Adds Up

Abstract

Vision–Language Models (VLMs) extend language models with an image encoder, but how visual tokens covering objects operate inside the LM backbone remains less understood than in text-only settings. Prior work has typically examined different forms of object-token influence separately, without comparing them under a common depth statistic. We study object-token influence in seven adapter-style VLMs along three operationally distinct axes—erasure sensitivity, cross-instance transfer, and category readout. Placing all three on a common half-mass depth coordinate, we find the same order in every model: erasure sensitivity precedes cross-instance transfer, which in turn precedes category readout. The two intervention axes concentrate in shallow-to-mid layers, whereas the readout axis concentrates in mid-to-deep layers, yielding an intervention–readout gap. Whole-set object-token interventions exhibit supra-additive effects relative to a matched background contrast; this excess concentrates in shallow layers and is indistinguishable from zero at deeper layers. These findings identify candidate depths for targeted intervention and motivate visual-token pruning criteria that account for joint rather than purely tokenwise effects.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language models, visual tokens, activation patching, causal intervention, representation readout, layerwise analysis, non-additivity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DDXrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how object-related visual tokens exert influence inside the language-model backbone of vision-language models. The authors operationalize object-token influence along three axes: erasure sensitivity, which measures how replacing object-token states affects the correct output; cross-instance transfer, which tests whether an object-token hidden state can causally transfer category-relevant information to another image; and category readout, which measures how well object category information can be recovered from the hidden representation. To compare these heterogeneous signals across depth, the paper converts each into a normalized layerwise mass profile and summarizes its location using a common half-mass depth statistic. Across seven adapter-style VLMs, the authors report a consistent depth ordering: erasure sensitivity concentrates earliest, cross-instance transfer later, and category readout deepest, yielding an intervention–readout gap. The paper further examines whether the effects of multiple object tokens combine additively, and finds that whole-set object-token interventions exhibit supra-additive effects relative to a matched background control, with this excess concentrated mainly in shallow layers and becoming indistinguishable from zero at deeper layers. Overall, the paper presents a systematic characterization of the depth organization and set-level interaction of object-token effects in VLMs, with potential implications for model intervention and visual-token pruning.

Critical strengths
  1. Extensive experiments and careful robustness analysis. The paper evaluates seven VLMs from five model families and includes a substantial set of controls and sensitivity analyses, including alternative baselines, prompt variations, threshold sweeps, pair-level statistics, and an LVIS extension. These analyses make the reported depth-ordering results reasonably well supported within the studied setting.
  2. A clear and unified framework for comparing different notions of object-token influence. The paper clearly distinguishes erasure sensitivity, cross-instance transfer, and category readout, and places these heterogeneous measurements on a common relative-depth coordinate using the same half-mass statistic. This provides a simple and interpretable way to compare where different forms of object-token influence concentrate across models.
Critical weaknesses

1. Limited scientific significance beyond descriptive characterization The paper carefully defines three operational notions of object-token influence and establishes a consistent depth ordering among them. However, the scientific story does not seem fully closed: it remains unclear what concrete model behavior this ordering explains, predicts, or enables. The paper does not connect the proposed influence profiles to downstream phenomena such as grounding, hallucination, robustness, or reasoning, nor does it show that the identified depths lead to better editing, intervention, or token pruning. The proposed applications to editing and pruning therefore remain largely speculative. Consequently, the work provides a strong answer to where these measurements concentrate, but a much weaker answer to why this organization matters or what can be done with it.

2. The category-readout axis does not establish object-token specificity The readout metric RR measures whether an object-token hidden state is more similar to text representations of its own category than to representations of other categories. This demonstrates category decodability from the selected token state, but does not establish that such information is specifically localized to object tokens. After contextual mixing, background tokens from the same image may also encode substantial information about the target object. A same-image background-token control—for example, comparing the similarity of object versus background tokens to the same category text—would help determine whether the reported readout is genuinely object-token-specific rather than reflecting more global image-level category information. Since the central Δ≺C≺R\Delta \prec C \prec R result interprets RR as an axis of object-token influence, this distinction directly affects the interpretation of the main claim.

Questions for authors
  1. What concrete model behavior is the proposed depth organization expected to explain or predict?
    The paper establishes a consistent Δ≺C≺R\Delta \prec C \prec R ordering, but it is not clear what observable VLM behavior this organization is intended to explain. Could the authors clarify the underlying scientific hypothesis behind these three notions of influence, and ideally test whether the resulting depth statistics correlate with or predict a concrete behavior such as object grounding, hallucination, robustness, or intervention effectiveness? Such evidence would help establish that the reported ordering is more than a descriptive regularity of the chosen measurements.

  2. Does the category-readout result remain object-token-specific when compared against same-image background tokens?
    The current RR metric contrasts the similarity of an object-token state to same-category versus other-category text representations. However, after contextual mixing, background tokens from the same image may also encode information about the target object. Could the authors add a control comparing object and background tokens with respect to the same target-category text, e.g., sim(hobj,ty)−sim(hbg,ty),\mathrm{sim}(h_{\mathrm{obj}}, t_y) - \mathrm{sim}(h_{\mathrm{bg}}, t_y), or a corresponding double contrast against negative-category text? More importantly, does the reported L50RL^R_{50}, and hence the central Δ≺C≺R\Delta \prec C \prec R ordering, remain qualitatively unchanged under such an object-vs-background-controlled readout?

  3. Can the authors demonstrate that the observed set-level non-additivity has a practical consequence for token selection or pruning? The paper motivates the non-additivity analysis partly through visual-token pruning, but currently only shows that the full-set intervention effect can exceed the sum of the two half-set effects. Could the authors test whether a tokenwise importance criterion actually makes worse pruning decisions at layers where JJ is large, compared with a group-aware criterion that accounts for joint effects? This would substantially clarify whether the non-additivity result has actionable significance rather than being only an interesting intervention phenomenon.

  4. How general is the full three-axis ordering beyond the specific COCO object-centric setting?
    The LVIS extension provides useful evidence for Δ≺C\Delta \prec C, but it does not re-evaluate RR or the set-level non-additivity result. Could the authors test the full Δ≺C≺R\Delta \prec C \prec R ordering, at least on a representative subset, under another dataset or a broader class of prompts/tasks? This seems important because the strongest conclusions currently rely on COCO segmentation-defined object tokens together with object naming and existence prompts, and it is therefore unclear whether the reported organization reflects a general property of VLM object processing or one specific to this experimental regime.

4 comments

Thank you for the detailed review. We ran the controls you asked for. Results are below and will be added to the paper.

W2 / Q2. Object-token specificity of R

We computed the double contrast you suggest, Rg,i′(ℓ)=Rg,i(ℓ)−Rgbg(ℓ)R_{g,i}'(\ell)=R_{g,i}(\ell)-R_{g}^{\text{bg}}(\ell), where RbgR^{\text{bg}} is Eq. (4) evaluated on same-image background states (patches no annotation touches). It contrasts object vs. background tokens and target vs. negative-category text at once.

CheckResult
Readout is object-specific (double contrast above zero)7/7
L50R′=L50RL_{50}^{R'}=L_{50}^{R}7/7 (0.472–0.765, as in Table 1)
L50C<L50R′L_{50}^{C}<L_{50}^{R'}7/7
G50CR′G_{50}^{CR'}, CI excludes zero (11-layer grid)6/7
Same, Qwen2.5-VL-3B on all 36 layers−0.056 [−0.083, −0.028]

So L50RL_{50}^R, and with it the ordering, is unchanged under the controlled readout. With the paper's θ\theta-based background set, the ordering also holds in 7/7. We use the double contrast because the single contrast sim(hobj,ty)−sim(hbg,ty)\text{sim}(h_{\text{obj}},t_y)-\text{sim}(h_{\text{bg}},t_y) also picks up a category-independent offset between object and background states. This offset is largest in the two Qwen models, where it alone changes the order (post hoc diagnosis). The negative-category term removes it. We will report both.

W1 / Q1. What behavior the ordering predicts

The hypothesis is that the depth at which category information is readable from object tokens is not the depth at which it drives the output. This predicts intervention effectiveness, which we tested. We transplanted the whole object-token set from a donor image at each model's transfer-peak layer and at its readout-peak layer, and measured how often the answer flips on held-out pairs.

  • The flip rate is higher at the transfer-peak layer in 7/7 models (difference +0.03 to +0.26, CI above zero).
  • At the readout-peak layer the flip rate is close to zero.

Layers selected by decodability are therefore layers where editing object tokens does not change the answer. For editing or steering, the profiles indicate which layers to use. We will add this experiment to §4.3.

Q3. Practical consequence for pruning

We ran the test on LLaVA-1.5-7B and Qwen2.5-VL-7B (200 pairs each) and added actual pruning on four models, using the attention-based rule of FastV (Chen et al., 2024) at a shallow layer near its default (ρℓ∈[0.08,0.11]\rho_\ell\in[0.08, 0.11]).

  • Tokenwise sums keep the ranking of token subsets (Kendall τ\tau 0.78–0.91, vs. 0.03–0.11 for attention scores).
  • Compared directly with a group-aware criterion at the same layers, tokenwise decisions are not worse (median regret 0). The joint effect does not change which tokens to prune.
  • Tokenwise sums do underpredict the size of the loss: at that layer, two half-set effects add up to 33–65% of the full-set effect.
  • In actual pruning, removing at least 90% of object tokens costs more than twice as much as removing about half (4/4 models, CI below zero).

The consequence is therefore for the pruning budget, not for the ranking. A tokenwise criterion picks the right tokens but underestimates the loss as more of an object is removed. We will add these results to §5.3.

Q4. Beyond COCO

We measured R on the LVIS pool of Appendix F, so the C–R ordering is now also evaluated there. R needs positive phrases in the text bank, which 247 of the 385 donor categories have (1,214 pairs).

  • L50C<L50RL_{50}^{C}<L_{50}^{R} with G50CRG_{50}^{CR} CI below zero: 6/7 models.
  • In Qwen2.5-VL-3B, C and R tie on the 11-layer grid. This is the same model for which the dense 36-layer run on COCO resolves the gap (above).

The set-level analysis remains on COCO. Its robustness is supported by the actual-pruning result in Q3 and by two checks on COCO: it holds in the lowest third of clean margins (7/7) and under re-drawn random half splits (two models).

W1 (significance), in short. The ordering is not only a description of where measurements concentrate. It separates layers where object tokens can be acted on from layers where they can only be read (Q1), and the set-level result changes how much pruning a tokenwise score can justify (Q3).

Thank you for the detailed and thoughtful rebuttal. The additional experiments have addressed most of my concerns, and I intend to revise my rating upward.

I have one remaining question regarding the independence of the new validation. Since the cross-instance transfer peak is identified through activation patching, and the new answer-flip experiment also relies on token transplantation, could the authors clarify how this experiment provides independent evidence of practical significance beyond the original transfer metric?

Thank you for the follow-up, and for reconsidering the rating.

You are right that both use transplantation, so the flip experiment shares its intervention family with the transfer axis. It is not circular, however, and its main conclusion does not depend on the transfer metric.

1. Out-of-sample. For each model we drew 300 pairs and split them in half. The transplant layers were chosen on the first half from the cached depth profiles, by a rule fixed before the run, and the flip rate was measured only on the second half. The flip outcome played no role in choosing the layers. The experiment therefore asks whether a layer fixed in advance predicts where an edit works on new images.

2. Different unit and outcome. The transfer metric is a continuous change in the naming margin under single-token transplants. The flip experiment transplants the whole object-token set and records whether the model's top-1 answer actually becomes the donor category. A margin shift does not imply an answer change, so this is a behavioral outcome that the transfer metric does not guarantee.

3. The readout side is independent of patching. The readout axis is a similarity between hidden states and category text, with no intervention. The practical question is whether the layers where the category is most decodable are layers where an edit works. As noted in our response, the flip rate at the readout-peak layer is close to zero: at most 0.007 in six of seven models. In InternVL3.5-8B the readout profile has two peaks of nearly equal height, at ρ=0.47\rho=0.47 and ρ=0.86\rho=0.86 (within 0.034), and the pre-specified argmax rule selected the shallower one, where the flip rate is 0.17. At the deeper peak the flip rate is 0.000, as in the other models. This result does not use the transfer metric: decodability is not a reliable guide for choosing where to intervene.

We will describe the experiment in §4.3 as an out-of-sample test of intervention effectiveness and state that it shares the intervention family with the transfer axis.

Thank you for the detailed and thoughtful clarification. I find the explanation convincing. I appreciate the authors' efforts and have decided to raise my rating.

Reviewer_UYooOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the influence of object tokens along three axes: erasure sensitivity, cross-instance transfer, and category readout. To enable a unified analysis, the authors first compute metrics for each axis across LLM layers, and then introduce a pair-level half-mass depth to provide a scale-free depth summary of the three axes. Based on the defined half-mass depth, the authors reveal that the three axes exhibit a consistent depth order across seven VLMs. In addition, a background-controlled evaluation is conducted to investigate the non-additivity of object tokens, showing that set-level non-additivity is concentrated at shallow depths.

Critical strengths

This paper jointly compares erasure sensitivity, cross-instance transfer, and category readout under a shared depth summary, providing useful insights into targeted intervention.

Critical weaknesses

The depth order of the three axes does not seem to be fully supported. Fig. 4 shows that Δ\Delta (erasure) comes early for all models, but C (transfer) is somewhat flat in Gemma-3-4B, Qwen2.5-VL-3B/7B, and InternVL3.5-8B. Moreover, R (readout) fluctuates significantly compared to Δ\Delta (erasure) and C (transfer). As a result, C and R sometimes overlap across the whole depth range, making their depth order not cleanly separable. This raises concerns regarding the validity of the key finding that the three axes show a consistent depth order.

Questions for authors
  1. The usage of background tokens in erasure analysis requires further justification. The paper uses raster order to replace object tokens with background tokens across LLM layers. However, the background tokens at deep layers may have already attended to object tokens, leading to object information leakage. This may be one reason why the half-mass depth of erasure sensitivity comes early. To ensure a clean evaluation, the authors may use background tokens that have not seen the objects to replace object tokens.
  2. Insufficient discussion of token pruning methods. The abstract mentions that the findings motivate visual token pruning criteria accounting for joint token-wise effects. However, the paper does not provide an in-depth discussion about this matter.
  3. The non-additivity analysis resembles erasure analysis. Fig. 3 shows that VLMs peak at shallow depths, which is correlated with the erasure effect. While it is interesting, the analysis does not provide new insights into the importance of object tokens.
4 comments

Thank you for the review. We answer each point below. New analyses will be added to the appendix.

W1. Separation of C and R

Fig. 4 shows cohort-mean curves, which average over pairs with different profile shapes. The ordering claim rests on pair-level half-mass depths (Eq. 5), not on these curves (Appendix C.3):

  • The paired gap G50CRG_{50}^{CR} is negative in 7/7 models (Table 9), and a Wilcoxon signed-rank test on h50,gC−h50,gRh_{50,g}^C-h_{50,g}^R is significant in 7/7.
  • πCR\pi^{CR}, the fraction of pairs with h50,gC<h50,gRh_{50,g}^C<h_{50,g}^R, is 56.7–91.0%.
  • Qwen2.5-VL-3B is the closest case: G50CR=−0.083G_{50}^{CR}=-0.083, one grid step. We re-measured it on all 36 layers: G50CR=−0.083G_{50}^{CR}=-0.083 [−0.111, −0.056].
  • With a readout that also contrasts object tokens against same-image background tokens, L50RL_{50}^R does not move in any model, so the order is unchanged in 7/7.

A flatter C profile still has a well-defined half-mass point. Flatness moves L50CL_{50}^C deeper, toward R, so it works against the ordering, which nonetheless holds. R fluctuates more because it is a similarity contrast, not a causal effect, and the pair-level statistics absorb this noise. We will add a pair-level version of Fig. 4 and state in the caption that the curves are cohort means.

Q1. Background tokens in the erasure analysis

We ran the control you propose in two forms. B~\tilde B: replacement states taken from the receiver image with the object removed. B′B': replacement states taken from a different image. In both cases the replacement tokens have never attended to the target object.

ControlShift of L50ΔL_{50}^{\Delta}Models
B~\tilde B (object removed)at most one grid step7/7
B′B' (different image)at most one grid step7/7

At deep layers (ρ≳0.65\rho \gtrsim 0.65) the difference between the original and the leakage-free erasure effect is indistinguishable from zero in 7/7 models. Leakage into background tokens therefore does not explain the early erasure depth.

Q2. Token pruning

We agree the paper should say more, and we now have results to base it on.

  • Ranking: the sum of single-token effects predicts the ranking of token subsets well (Kendall τ\tau 0.78–0.91), far better than attention scores (0.03–0.11); LLaVA-1.5-7B and Qwen2.5-VL-7B, 200 pairs each.
  • Magnitude: the same sum underpredicts the loss from joint removal. At a shallow layer near where FastV (Chen et al., 2024) prunes (ρℓ∈[0.08,0.11]\rho_\ell\in[0.08, 0.11]), the two half-set effects add up to only 33–65% of the full-set effect (ratio of means).
  • Actual pruning: when FastV's attention-based rule is applied at that layer and pruned tokens are masked from keys and values in all later layers, the loss from removing at least 90% of object tokens is more than twice the loss from removing about half (4/4 models: LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3.5-8B, Gemma-3-4B; CI below zero in each).

So tokenwise scores are reliable for deciding which tokens to drop first, but not for deciding how many can be dropped. The loss grows faster than tokenwise scores predict once most of an object is gone. We will add this discussion to §5.3.

Q3. Non-additivity vs. erasure

The two are correlated, as they must be: an interaction can only appear where erasure has an effect. They are not the same quantity, and the interaction is not a rescaled erasure effect:

  • Normalized by erasure size, IX/∣δXfull∣I^{X}/|\delta_{X}^{\text{full}}| is larger for the object set than for the background set in 7/7 models (difference +0.24 to +0.40; background median near zero).
  • Two object halves account for 60–80% of the full-set effect (pair median over the shallow band). For background halves the ratio is 0.94–1.60.
  • The result holds in the lowest third of clean margins (7/7), so it is not a saturation effect of large erasures.

What it adds about the importance of object tokens: erasure says the object set matters, and non-additivity says how that importance is distributed inside the set. Half of the object tokens carry most of the evidence by themselves, so single-token or half-set importance understates the set. This is the property that the pruning result in Q2 depends on. We will revise §5.3 to state this relation to erasure explicitly.

Thanks for the detailed response and additional experiments. Some of my concerns have been addressed, and a few points remain for discussion.

W1. Depth order of the three axes.

Rebuttal: "The ordering claim rests on pair-level half-mass depths (Eq. 5), not on these curves."

The pair-level statistics support the ordering for typical pairs. However, the ordered fraction for C ≺ R is as low as 56.7%, and the gap for Qwen2.5-VL-3B is only a single grid step. Therefore, while the ordering is statistically consistent, the axes are not cleanly separated, which was my original concern. Since half-mass depth summarizes location rather than spread, it would be helpful to report a spread measure to show the extent of overlap among the axes. In addition, framing the finding as a shift in where mass concentrates, rather than a clear separation, seems more appropriate.

Q3. Non-additivity vs. erasure.

Rebuttal: "Half of the object tokens carry most of the evidence by themselves, so single-token or half-set importance understates the set."

The normalized comparison and low-margin check show that non-additivity is not simply a rescaled erasure effect, and the connection to pruning makes the analysis more useful. However, since redundancy among visual tokens has also been reported in prior pruning work, it would be helpful to clarify what additional insights the background-controlled measure provides beyond existing findings.

Thank you for the follow-up. We ran the analysis you suggested.

W1. Depth order and overlap of the axes

For each pair we computed the interquartile depth range [q25, q75] of its normalized influence mass. The pairs are the same as in Table 9, and q50 reproduces the reported L50 exactly.

ΔCR
Median width (q75 − q25), relative depth, range over 7 models0.19–0.310.28–0.390.31–0.47

For C vs R, the two ranges of the same pair are fully separated in 4–33% of pairs, and the median intersection-over-union is 0–0.36 across models. In most pairs the two ranges are adjacent or partially overlapping, with C's range placed shallower than R's. Results against R′, computed on the pairs where R′ is defined, are nearly identical.

We agree with the framing you suggest. We will describe the result as a consistent shift in where influence mass concentrates within a shared depth range, and we will add the spread statistics with per-model tables to the appendix.

Q3. What the background-controlled measure adds to pruning results

Pruning studies infer redundancy from task accuracy after tokens are removed. Our measure adds three things.

  1. Specificity. The two object halves together account for 60–80% of the full-set effect (pair median), while for background halves the same ratio is 0.94–1.60. The redundancy belongs to the object evidence, not to visual tokens in general.
  2. Depth. The supra-additivity is confined to the shallow band and is indistinguishable from zero in deep layers. How redundant a token set is depends on the layer at which it is removed.
  3. Magnitude. Single-token scores preserve the ranking of tokens (Kendall τ 0.78–0.91) but understate the effect of removing them together: the sum of the two halves is 33–65% of the joint effect at a shallow layer near where FastV prunes (ratio of means). In FastV-style removal, the loss accelerates once most object tokens are removed, in all four models tested.

We will add this comparison to the discussion of token pruning.

Thanks for the additional analysis. The spread statistics show that the C and R ranges are fully separated in only 4–33% of pairs. The authors also acknowledge that the results are better characterized as a shift in where influence concentrates within a shared depth range. However, the consistent depth ordering is a central claim of the submission, emphasized in both the abstract and the contributions. Reframing this finding as a shift in influence concentration substantially weakens the main claim and would require reassessing the paper’s contribution, which goes beyond what can reasonably be addressed in a camera-ready revision.

Overall, I appreciate the authors’ engagement, but I think the current version may not be ready for publication.

Reviewer_gZYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates two effects of object tokens in VLMs: the depth organization and set-level non-additivity, then identifies a three-axis depth ordering and shallow background-relative supra-additivity. The experiments span seven VLMs with several controls. However, the findings remain most descriptive. Insights or practical implications are generally limited. Furthermore, the overall methodology is relatively straightforward, and findings reported by this paper appear more incremental than substantially novel to me.

Critical strengths

Systematic empirical evaluation. Multiple VLM families are evaluated in the proposed framework. Carefully designed intervention controls and robustness analyses are also included in experiments. Overall, the experiments are well designed and make the findings more convincing.

Critical weaknesses
  1. Limited insights and practical implications. The authors report empirical patterns but do not go much further into why they occur or how the behaviors of VLMs could be connected to them. Applications suggested in conclusion part are not extended, making what this research could bring to the research community unclear.

  2. Limited novelty in methodology and findings. This study mainly combines intervention and readout techniques in a relatively straightforward way. While the cross-model ordering and shallow non-additivity provide some new empirical evidence, the overall contribution seems incremental.

Questions for authors
  1. Reported positive background-controlled non-additivity (J>0J > 0) does not necessarily imply supra-additivity of the object-token set Iobj>0I_{\mathrm{obj}} > 0. Could the authors report IobjI_{\mathrm{obj}} clearly and clarify how the results support your broader non-additivity claim?

  2. What new understanding of VLM behavior can the findings provide, beyond the reported empirical patterns?

4 comments

Thank you for the review. We answer each point below. All new analyses will be added to the appendix.

Q1. IobjI^{\text{obj}} on its own

The claim holds without the background term. Shallow-band means (ρℓ≤0.30\rho_\ell\le0.30), with the same two-way cluster bootstrap as Table 10:

ModelIˉobj\bar I^{\text{obj}} [95% CI]Iˉbg\bar I^{\text{bg}} [95% CI]
LLaVA-1.5-7B+0.30 [+0.18, +0.46]−0.01 [−0.02, +0.00]
LLaVA-1.5-13B+0.24 [+0.14, +0.38]+0.00 [−0.01, +0.01]
Idefics3-8B+0.21 [+0.15, +0.29]+0.03 [+0.01, +0.05]
Gemma-3-4B+0.90 [+0.46, +1.53]−0.01 [−0.05, +0.03]
Qwen2.5-VL-3B+0.20 [+0.14, +0.28]−0.03 [−0.07, +0.01]
Qwen2.5-VL-7B+0.57 [+0.37, +0.85]−0.12 [−0.17, −0.07]
InternVL3.5-8B+0.82 [+0.52, +1.21]−0.01 [−0.05, +0.03]

Iˉobj\bar I^{\text{obj}} is positive with a CI above zero in 7/7 models. Although the background arm itself moves the margin in several models (Table 11), its non-additivity term Iˉbg\bar I^{\text{bg}} is small: at most 21% of Iˉobj\bar I^{\text{obj}} in magnitude, and its CI includes zero in five models. So Jˉband>0\bar J_{\text{band}}>0 comes from the object set.

At the two deepest grid layers Iˉobj\bar I^{\text{obj}} is indistinguishable from zero in 7/7 models. Two further checks: the result holds in the lowest third of clean margins (7/7), so it is not margin saturation, and it is stable under re-drawn random half splits (re-run on two models). We will report IobjI^{\text{obj}} and IbgI^{\text{bg}} separately in §5.2.

W1 / Q2. What the findings tell us about VLM behavior

  1. Where category information can be read is not where it can be changed. By the depth at which half of the readout mass is reached, 74–89% of the erasure mass and 49–80% of the transfer mass has already accumulated. We tested the behavioral consequence directly: transplanting the whole object-token set flips the answer more often at the transfer-peak layer than at the readout-peak layer in 7/7 models (held-out pairs, CI above zero), and at the readout-peak layer the flip rate is close to zero. Choosing an editing layer by decodability therefore selects layers where edits on object tokens no longer act.

  2. Object evidence is spread redundantly over object tokens. Erasing half of the object tokens removes only about a third of the full-set effect. The two halves together account for 60–80% of it, while for background halves the same ratio is 0.94–1.60. Per-token or per-subset importance therefore understates what a set of tokens carries. This also shows up in actual pruning. We applied the attention-based rule of FastV (Chen et al., 2024) at a shallow layer near its default (ρℓ∈[0.08,0.11]\rho_\ell\in[0.08, 0.11]), masking the pruned tokens from keys and values in all later layers. The loss from removing at least 90% of object tokens is more than twice the loss from removing about half, in 4/4 models tested.

  3. Why it occurs. The interaction is already present when erasure is applied at the input of the first LM layer (7/7, at 81–100% of its band peak) and stays flat through the shallow band, so the early LM layers do not create it. Blocking all attention among visual tokens in every LM layer keeps 41–85% of it (three models, CI above zero in each). Mixing among visual tokens inside the LM is thus only part of the cause. The remainder does not depend on such mixing.

We will add these three points to the ends of §4.3 and §5.3.

W2. Novelty

The individual tools are standard on purpose, so that each measurement is easy to verify. What is new is the comparison they allow:

  • Erasure, transfer and readout are measured on the same image pairs, on one depth coordinate, with pair-level statistics, across seven models from five families. Earlier studies typically report one kind of probe at a time, which leaves open whether different depth conclusions come from the models or from the probes.
  • A direct test of whether object-token effects add up, with a matched background control. To our knowledge this has not been reported for visual tokens in VLMs. The redundancy literature infers it from pruning accuracy, not from a set-level interaction measured by depth.
  • The intervention–readout gap survives an object-vs-background readout control (order kept in 7/7 models) and an extension to LVIS (6/7).

Dear Reviewer gZYD,

Thank you again for your thoughtful review. We hope our response has addressed your concerns, in particular Q1 (the object-set term reported on its own, positive in 7/7 models) and W1/Q2 (what the findings tell us about VLM behavior).

If any point still feels unresolved, please let us know. We would be happy to clear it up while the discussion period is still open.

Thank you for the detailed rebuttal and additional experiments.

The newly reported results on object-set non-additivity directly address my first question, and the answer-flip experiments provide useful behavioral evidence for the intervention–readout gap. I appreciate these additions and consider my first concern resolved. However, I still find the overall contribution somewhat limited. The new behavioral results would benefit from more detailed quantitative comparisons, while the pruning experiments do not clearly establish the practical value of the observed non-additivity. Moreover, observing non-additivity when intervening at the LM input does not necessarily establish that the early LM layers do not create it, since the effect is measured at the final output.

Overall, although the rebuttal addresses some of my concerns, I will maintain my original score.

Thank you for reading the rebuttal and for confirming that Q1 is resolved. Brief replies to the three remaining points.

Input-layer result. You are right, and we will correct the wording. Since the effect is read at the output, applying erasure at the LM input cannot show where the interaction is computed. It supports only a narrower statement: the interaction is nearly the same size wherever in the shallow band the erasure is applied, including at the LM input. The result that bears on mechanism is the attention-blocking one. With all attention among visual tokens blocked in every LM layer, 41–85% of the interaction remains (three models, CI above zero in each), so that part does not require visual tokens to exchange information inside the LM. Where in the remaining computation it arises is open. We will say this in §5.3 in place of "the early LM layers do not create it".

Behavioral result. The table below gives per-model flip rates, i.e. how often the top-1 answer becomes the donor category after transplanting the whole object-token set. Layers were fixed in advance on one split of 150 pairs, and rates are measured on the other 150 (149 for Idefics3-8B).

ModelTransfer-peak layerReadout-peak layerDifference [95% CI]
LLaVA-1.5-7B0.150.007+0.14 [+0.09, +0.20]
LLaVA-1.5-13B0.130.000+0.13 [+0.07, +0.18]
Idefics3-8B0.180.007+0.17 [+0.11, +0.23]
Gemma-3-4B0.260.000+0.26 [+0.19, +0.33]
Qwen2.5-VL-3B0.030.000+0.03 [+0.007, +0.05]
Qwen2.5-VL-7B0.110.007+0.10 [+0.05, +0.15]
InternVL3.5-8B0.370.17+0.20 [+0.14, +0.27]

At the readout-peak layer at most one pair flips in six models. In InternVL3.5-8B the readout profile has two peaks of nearly equal height and the pre-specified rule selected the shallower one; at the deeper peak the flip rate is 0.000. This table will go into §4.3.

Pruning. We do not propose a pruning method, and the claim is limited to the pruning budget. Tokenwise scores rank token subsets well (Kendall τ 0.78–0.91, two models), so non-additivity does not change which tokens to drop. It changes how much loss to expect: at a shallow layer near where FastV prunes, the two half-set effects sum to 33–65% of the full-set effect, and in FastV-style removal the loss from removing at least 90% of object tokens is more than twice the loss from removing about half (4/4 models). We will state the implication in the paper in these terms.

Comments on the paper

1 comment
Area_Chair_sVsDOpenReview ↗

Please check the authors' rebuttal ASAP.

Dear reviewers,

As the discussion period will end very soon, please check the authors' rebuttal and see if your concerns have been addressed. Thank you.

-AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.