- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies attention-mediated information flow in vision transformers by placing a variational bottleneck on each attention head’s update before it is written to the residual stream. The authors train separate ViT-Tiny models with different information penalties and examine how accuracy, patch-level predictions, and representations change as the constraint is relaxed. Under tight constraints, only a few heads remain active. The paper analyzes these heads and finds that some respond to repeated patches within an image. It also applies the same approach to DINO training.
Critical strengths
- The paper is well written and easy to follow.
- The motivation is reasonable. Restricting attention writes during training offers a way to study simpler models whose internal computations may be easier to inspect.
- The copy-paste experiment is a useful step beyond visualizing attention maps. It provides evidence that selected heads respond to repetition elsewhere in the image while the queried patch remains unchanged
Critical weaknesses
- This paper treats information retained in a head update as a measure of communication between patches. However, the update can also contain information about the receiving patch itself. Neither the KL cost nor the reported mutual information cleanly isolates information arriving from other patches. This makes the paper’s central interpretation less precise than its language suggests.
- The repetition-sensitive heads are an interesting finding, but their role in the model’s predictions is not established. The copy-paste experiment shows that their messages respond to repeated content. It does not show whether those messages are needed for classification, or whether an unrestricted ViT uses a comparable computation.
- The most detailed head-level analysis concerns strongly constrained ViT-Tiny models on ImageNet-100. It is unclear how well these findings carry over to less constrained or larger models. The bottlenecks also add substantial parameters, and removing the KL penalty does not fully recover the unmodified ViT’s accuracy.
Questions for authors
- How much of the measured information in a head update comes from other patches rather than the receiving patch?
- What happens to predictions when a repetition-sensitive head is disabled? Is similar behavior present in an unrestricted ViT?
- Do the head-level findings persist with a larger backbone or a more varied dataset?
We appreciate the feedback on our submission, which led to some exciting improvements to the manuscript.
W1/Q1, cross vs local information: The attention output is what’s restricted, and it’s a function of all patches (including the receiving patch). When the information that attention writes -- the only way information can pass between patches -- is restricted, then total cross communication is restricted. We found several places where our prose could be misinterpreted and revised them to be more clear to this point. The guiding intuition is that under information restriction, processing of the receiving patch's information will be done by the MLPs where it is not penalized. The attention heads will then add only essential updates that are themselves a function of both the receiving patch and the rest of the image (as in the repetition detection, which is necessarily a function of both).
W2/Q2, role of repetition head in prediction: If ablated, the accuracy matches that of the local patch processing, 70% -> 69%. Thus with just one head active and about 1 nat KL for every 10 patches, we get a 1% boost and about a tenth of the total effect of having an unrestricted ViT, which is remarkable. Thanks to the simplicity of the information-starved models, we have other interventions as well: we can steer in the latent space of the active head’s information bottleneck, which is effectively one-dimensional. The results are interesting; please take a look at our new Fig. 5 if you have a chance. In short, while the head's channel is ~simple, the effect on downstream prediction is highly patch dependent. We looked and did not find strongly matching behavior in existing ViTs (by correlating attention maps), nor was it particularly present in ViT-IB models with more communication. We assume that with less/no pressure toward economical messages, the behavior is smeared across many heads.
W3/Q3 larger backbone, more varied dataset: For the revision, we have also trained a series of ViT-S and Vit-B backbones; intriguingly, we found similar repetition and foreground/background detection heads in their lowest-info models too. In terms of parameter count, the relative size of the VIB modification shrinks from +40% params for ViT-tiny to only +10% for ViT-B. We recover a reasonable spectrum of accuracy, similar head sparsification, and head behavior in both backbones that matches the repetition and the foreground/background heads in the ViT-tiny series.
W3 removing the penalty does not fully recover ViT accuracy: This is a fair point. The boost in expressivity from the added MLP layers appears to be more than offset by breaking the linear structure of the writes to the residual stream. With a biased opinion, to be sure, we think the added interpretability is worth the cost, though we agree that this is an encumbrance to shed in future work.