- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the causal anatomy of ViT classification. It finds that class evidence develops within the patch stream and is written into [CLS] token only through a few terminal layers, called the read window. Motivated by this, the paper introduces RTLens, which reads intermediate patch states through the read window rather than directly through the final head, and an attention map that attributes the prediction margin to messages delivered through that window.
Critical strengths
-
The paper is in general easy to follow.
-
The proposed principle "read where the model reads", is simple and compelling. RTLens directly follows the measured causal path and improves over the naive logit lens across multiple ViT backbones.
-
The paper presents some clear and interesting findings supported by empirical evidence.
-
The experimental evaluation is careful and broad. The paper studies five ViT-Base backbones, includes a ViT-Large scale probe, uses distribution-preserving interventions, and provides extensive study in the appendix.
Critical weaknesses
-
The paper is strong overall, but the main causal anatomy is still fairly coarse. The analysis identifies important token-group routes such as P←P and C←P, but it does not resolve the mechanism at the level of individual heads, MLPs, features, or circuits. The authors acknowledge that the two-stage anatomy is not the complete classification circuit, but many of the claims are phrased quite strongly around “where the model reads". In practice, the results show the dominant token-group communication path, but not a full mechanistic explanation of how class evidence is represented, transformed, or selected.
-
The qualitative examples are helpful but somewhat selective. For the patch-label maps and attribution maps, more randomly sampled examples, failure cases, and side-by-side comparisons across model families would make it easier to understand when the method works well or poorly.
-
The paper does not fully explain why different models have different read-window structures. DINOv2, DINOv3, and CLIP are dominated by the final layer, while supervised ViT and DeiT use a two-layer read, and ViT-L appears to use a wider terminal region. The empirical finding is interesting, but it would be good to provide a deeper explanation of what architectural or training factors determine the width and timing of the read.
-
The paper's conclusions seem to be tied to CLS-based classification readouts only. The results in appendix show that under mean-pool readout on DINOv2, blocking P← P is highly damaging while blocking C ←P is almost irrelevant.
Questions for authors
See Weaknesses.
Follow-up on Our Rebuttal Responses
Dear Reviewer Vr3s,
Thank you again for taking the time to review our paper and provide constructive suggestions.
Following your suggestions, we have conducted additional experiments on CaiT and a direct comparison between RTLens and JLens. We have incorporated the results and their interpretations into our rebuttal responses and revised manuscript.
With approximately 25 hours remaining in the discussion period, we would appreciate it if you could review our responses and share your thoughts before the deadline.
Best regards,
The Authors