- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies decoder-written visual massive activations in 25 adapter-based vision-language models. Ten exhibit spikes under the measured criterion. The authors identify inherited blocks and trigger directions associated with spike formation, relate eventual spike locations to the least-shared image tokens at decoder input, and demonstrate sensitivity to corruptions and small adversarial perturbations. Removing the trigger component suppresses spikes, with limited task-score changes in most models. The paper leaves their functional role unresolved.
Critical strengths
The distinction between channel concentration, massive activation and attention sinks is well supported and matters for interpreting earlier measurements. The breadth of the model survey is valuable, as are the comparisons with the corresponding text-only bases. Trigger interventions, state swaps and random-direction controls provide substantially more mechanistic evidence than a correlation study. The authors also report exceptions and attention-implementation sensitivity rather than presenting a uniform account unsupported by the measurements.
Critical weaknesses
-
The experiments do not yet establish the value of the learned controller over simpler adaptation. The paper does tune a global positive coefficient on validation data, but that does not replace a comparison with a simple gate using one tuned signed coefficient. Such a control would distinguish the value of selecting when to intervene from that of learning a 40-bin magnitude distribution. The headline 24-point improvement on LLaVA VOC is also achieved by VCD: both reach CHAIRi 14.0. It is a valid comparison with Regular decoding, but does not establish the added value of TAVCD.
-
The deployment features need clarification. Table 2 defines t/T using the total caption length, which is unavailable before generation terminates. If inference substitutes a maximum generation length or an estimate, the paper should specify it and use the same definition when training the predictor. As written, the online feature computation cannot be reproduced unambiguously.
-
The oracle labels provide a weaker guarantee than the term “structurally uncorrectable” suggests. Appendix B searches a bounded, discrete grid on the stored top-50 support; failure there does not establish that the full real-valued feasible set is empty. A C3 violation excludes positive coefficients, not necessarily negative ones.
-
The recall tradeoff is substantial for BLIP-2 on VOC: 61.6 with VCD falls to 49.0 with TAVCD. CHAIRi improves from 16.4 to 16.2 and CHAIRs from 18.0 to 14.2, but these reported gains accompany considerably less object coverage. Object-mention counts and caption lengths would help characterize this tradeoff.
-
The reported “less than 1%” overhead concerns the head, not the two-branch decoder. End-to-end latency against Regular decoding and VCD is still missing.
Questions for authors
Where are the residual spikes written in InternVL3.5-8B after trigger removal, and do they involve the same tokens and blocks as before intervention?
2 comments
Dear Reviewer Mjev,
We appreciate your initial positive feedback. We would be happy to provide any further clarification before the discussion period ends.
Thank you for your time and consideration.
Thank you for reviewing our paper. We appreciate your recognition of the breadth of our study, the mechanistic evidence provided by our experiments, and our candid reporting of model-specific exceptions. We address your question below.
Residual spikes after trigger removal
We appreciate the reviewer’s thoughtful question, which led us to investigate the residual spikes further.
In InternVL3.5-8B, every residual spike after trigger removal is written at the output of the visual switch block, the same block that writes the clean spike, and in all 66 forward passes with residual spikes, the spike occurs at the same image token as in the clean forward. We extended the analysis to the four models where the trigger intervention leaves residual spikes. The same token spikes in 6 of 6 residual cases in InternVL3.5-4B, 25 of 31 in InternVL3.5-14B, and 86 of 91 in Molmo2-8B, and every residual spike peaks at the switch's output. We found that the residual spikes arise from the other dominant eigendirection of the switch, the one of the opposite sign, which becomes prominent once the trigger direction is projected out. Projecting out this second direction as well eliminates them completely.
Based on this finding, we have updated our intervention to project out both the trigger t and t−, obtained by running the trigger search with the write sign reversed. In these four models, t matches v+ and t− matches v−, the eigenvectors associated with the largest positive and most negative eigenvalues of the switch's quadratic form, respectively. Removing both directions eliminates all residual spikes, without changing our overall results and conclusions (Table 1; ablation in Table 50).
Table: Origin of residual spikes and effectiveness of the updated intervention. We re-evaluate the spiking POPE forwards from Table 1, first projecting out t at the visual switch, then projecting out both t and t−. Every residual spike peaks at the output of the original visual switch block. Values leaving the switch are medians over forwards with residual spikes; the cosine similarity between t and v+ is at least 0.999.
! The critical weaknesses section appears to refer to another paper. Please let us know if anything about our work needs clarification.
Revisions to the manuscript are highlighted in blue.