- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that a ViT attention head should not be treated as a single homogeneous attention map. Instead, the authors analyze the joint query–key (QK) feature, i.e., the concatenation of a query vector and a key vector, and organize it into a hierarchy. The pipeline is: (1) record QK features from above-threshold attention pairs, (2) recursively cluster them with UMAP + HDBSCAN to build a per-head "QK tree", and (3) interpret each node via a "role card" that visualizes Q-side/K-side frequency maps, prototype images, and spatial displacement statistics.
The paper supports this with four sets of evidence: (i) QK features are multi-modal rather than a single cloud, validated against shuffled / covariance-matched / dimension-permuted / random nulls (Table 1); (ii) nodes at greater tree depth show higher semantic agreement, measured with a DINO-probe weighted-mean-direction statistic against a parent-matched null (Figure 3); (iii) role cards expose interpretable behaviors, including position-indicator nodes concentrated in early layers; (iv) selecting a single object-relevant node yields better saliency IoU than best-head or best-layer baselines on ECSSD / DUT-OMRON / DUTS-TE (Table 2). A causal knockout experiment (Appendix G) shows node removal perturbs the output more than size-matched random controls.
Critical strengths
-
The QK-feature framing and the hierarchical decomposition are a genuine methodological contribution. Prior head-level work either aggregates attention maps or projects head contributions into an external embedding space. Directly partitioning the joint (q, k)distribution and validating multi-modality against several structured nulls (Table 1, Table 4) is a cleaner and more falsifiable approach. This matters because it makes the "attention head is not one thing" claim testable rather than rhetorical.
-
The depth-dependent semantic coherence result is the strongest empirical finding. The monotonic increase of r with depth, replicated across four backbones, both Q and K sides, four layer groups (Figure 13), and two prototype counts k=10 and k=50 (Figure 14), and with the parent-matched null strictly dominated in all 24 cells, is a well-controlled result. It supports the central claim that deeper nodes are more semantically specific.
Critical weaknesses
-
The localization result is the paper's main "actionable" claim, and it is the least clean part. Node selection uses 500 labeled DUTS-TR masks, while the method is presented as exposing structure that coarser aggregation obscures. The gains over best-head/best-layer are large and consistent (Table 2), so the empirical result stands, but the paper should either (a) present a label-free selection criterion, e.g., using the §5.2 semantic-consistency statistic or the position-indicator score, and report how much of the gain survives, or (b) explicitly reframe the contribution as "node-level selection given a small calibration set" rather than as a training-free localization method. As written, the abstract and introduction oversell the degree of supervision-freeness.
-
The semantic-consistency evidence and the localization evidence are not connected. The paper argues that nodes carry consistent semantic roles and separately that one node localizes objects well, but it never checks whether the node selected by IoU is also the node with high r ̅_v^Q or a coherent role card. If the two criteria selected different nodes, the "semantic role" interpretation of the localization node would be undermined. A single correlation or case study linking the two would substantially strengthen the internal coherence of the paper.
Questions for authors
-
In Table 2, the node-level method uses a fixed threshold of 0.5 while the head and layer baselines are allowed to choose among seven binarization rules. Could the authors report node-level results under the same rule-selection protocol, or alternatively report head/layer baselines at the fixed 0.5 threshold, to confirm the comparison is not driven by threshold flexibility?
-
For the localization experiment, what happens if node selection uses only unlabeled criteria, e.g., ranking nodes by r ̅_v^Qor by position-indicator MI, on the same 500 DUTS-TR images without ground-truth masks? How much of the IoU gain over best-head is retained?
-
Is the node chosen for localization in each backbone also among the nodes with the highest semantic agreement or the most coherent role card? A brief case study would clarify whether the two analyses identify the same structure.
-
How many heads actually produce a tree of depth ≥ 2 under the main-paper configuration (s_"gate" =0.5, d_max=3)? The Dip-means splittable fractions in Table 1 are computed with a different procedure than the main pipeline, so the reader cannot directly tell what fraction of heads contribute non-trivial hierarchies to Figures 3 and 4.
-
- In Appendix G, the depth-2 ViT result loses significance when restricted to nodes covering at least 0.1% of the attention matrix. Does the paper's claim that deeper nodes are more functionally specific survive this restriction in the other backbones, or is the causal evidence effectively limited to depth-1 nodes?
7 comments
Weakness 2
W2. Linking semantic consistency to the localization node
The semantic-consistency evidence and the localization evidence are not connected. The paper argues that nodes carry consistent semantic roles and separately that one node localizes objects well, but it never checks whether the node selected by IoU is also the node with high \bar{r}^Q_v or a coherent role card. If the two criteria selected different nodes, the "semantic role" interpretation of the localization node would be undermined. A single correlation or case study linking the two would substantially strengthen the internal coherence of the paper.
Thank you for the comment. First, we clarify that the localization mask used in §7 is the Q-frequency map of each node v, and that in §5 is computed from the same frequency map. That is, the two analyses concern the same underlying map but evaluate different properties of it: (a) whether the map lands on the same semantic part across images (), and (b) whether that part overlaps with the entire salient object (IoU).
Therefore, the two criteria need not select the same node, as they measure different properties of the same frequency map. measures part-level consistency. For example, a node that focuses on a narrow part, such as the "from the animal's head to its ear" node in Figure 4, has a high , but since that part is a small portion of the saliency mask, its IoU is low. Conversely, a node that covers the entire object can achieve a high IoU without necessarily having the highest . The claims of the paper are that "the nodes of the QK tree have (statistically) consistent semantic roles" (§5) and that "selecting just one node among them with an object-level role already localizes better than heads and layers" (§7), not that "the most consistent node is the best for localization."
Nevertheless, we agree that a case study is useful, and we have added to Appendix H, for each backbone, the role cards of the two nodes side by side (Figs. 64–67): the node with the highest , as in Section 5 (top), and the node selected in Table 2, as in Section 7 (bottom). The top- nodes respond to the sky (ViT), the background surrounding the object (CLIP), or the image border (DINOv2, DINOv3); they land at the same place in every image but are unrelated to the object. In contrast, the selected nodes are object-level nodes whose frequency maps cover the foreground object across all four backbones, with query–key displacements concentrated at (0, 0). Both nodes have consistent role cards; the difference is "what" they respond to consistently. This explains why and IoU select different nodes, while the role cards provide qualitative evidence that the IoU-selected nodes also exhibit coherent semantic roles.
Questions 1, 2, and 3
Q1. Same binarization protocol for nodes and head/layer baselines
In Table 2, the node-level method uses a fixed threshold of 0.5 while the head and layer baselines are allowed to choose among seven binarization rules. Could the authors report node-level results under the same rule-selection protocol, or alternatively report head/layer baselines at the fixed 0.5 threshold, to confirm the comparison is not driven by threshold flexibility?
First, we would like to clarify that this asymmetric protocol was a deliberate design choice. The [CLS]-to-patch attention map used by the head/layer baselines is a probability distribution that sums to 1, whereas the node's is a frequency map with values in [0, 1] that is not normalized to sum to 1. Because the two maps have different normalization properties, applying the same numerical threshold would not yield an equivalent binarization criterion. We therefore deliberately made the head/layer baselines stronger by letting them choose the best of seven binarization rules, while fixing the node to the single value of 0.5, to test whether node-level selection still outperforms head- and layer-level selection under this less favorable condition. In fact, the map-mean rule was selected in 7 of the 8 baseline settings (App. J), so the baselines in Table 2 already use the rule selected as optimal on the calibration set.
Nevertheless, for reference, as the reviewer suggested, we present below the results obtained by removing the calibration-based rule selection for the head/layer baselines and fixing them to a single rule, cumulative-mass 0.5. Everything else is identical to the paper. As the table shows, removing this additional flexibility lowers the best head/layer by .07–.21, and the node's advantage becomes larger.
| ECSSD IoU | ViT | CLIP | DINOv2 | DINOv3 |
|---|---|---|---|---|
| Best head, fixed mass 0.5 (Table 2: best of 7 rules) | .370 (.451) | .287 (.373) | .429 (.535) | .357 (.472) |
| Best layer, fixed mass 0.5 (Table 2: best of 7 rules) | .366 (.438) | .164 (.375) | .421 (.500) | .373 (.485) |
| Ours (one node), fixed 0.5 (i.e., Table 2) | .469 | .569 | .647 | .682 |
Q2. Label-free node selection ( or position-indicator MI)
For the localization experiment, what happens if node selection uses only unlabeled criteria, e.g., ranking nodes by \bar{r}_v^Q or by position-indicator MI, on the same 500 DUTS-TR images without ground-truth masks? How much of the IoU gain over best-head is retained?
Thank you for the question. Selecting nodes by yields the sky, background, and image-border nodes shown in Figs. 64–67 of W2. Their calibration IoU is already below 0.02, far below that of the best head, so none of the gain over the best head would be retained. The position-indicator MI falls below the threshold for almost all nodes beyond layer 2, making it unsuitable for reliably distinguishing nodes in layers 6–11 (App. I). For further discussion, please see our responses to W1(a) and W2.
Q3. Overlap between the localization node and high-consistency / coherent role-card nodes
Is the node chosen for localization in each backbone also among the nodes with the highest semantic agreement or the most coherent role card? A brief case study would clarify whether the two analyses identify the same structure.
Please see our response to W2 for a detailed discussion.
Weakness 1
W1. Label-based node selection vs. training-free framing
We understand the reviewer's point. We answer (a) and (b) in turn.
(a) Label-free selection criterion
First, we would like to clarify the purpose of §7. §7 does not propose a label-free localization method; rather, it shows that, under the same frozen attention and calibration set, selecting QK nodes, a finer-grained unit than heads or layers, is more effective than selecting entire heads or layers. For this reason, rather than devising a label-free selection criterion for each of the three methods (layer, head, node), we compare them under the same calibration setting.
The two criteria suggested by the reviewer are difficult to use as label-free selection criteria. (1) rˉQ measures whether a node lands at the same place across images, not whether that place is an object. (We discuss this in more detail in W2.) (2) The position-indicator MI, as shown in App. I, is concentrated in layers 0–2, and almost all leaves beyond layer 2 fall below the threshold, so it cannot distinguish nodes in layers 6–11; moreover, selecting nodes that respond to fixed positions would favor spatial consistency rather than object localization, which must follow the object.
Therefore, designing and validating a label-free selection pipeline is beyond the scope of the present work. That said, we consider the direction the reviewer points to, namely finding object-centric nodes using only the statistics of the QK tree, to be very promising future work, and we see it as our next task. Accordingly, we have chosen the reframing of (b) for this revision.
(b) Reframing our contribution
We would like to clarify that the paper does not claim that our localization method is label-free, and that our use of "training-free" refers only to the absence of backbone parameter updates. Of the five occurrences of the term training-free in the paper, four refer to prior work (e.g., object discovery based on DINO attention), and the remaining one, in App. A (Limitations), already states explicitly that our method is "training-free in the sense that no backbone parameters are updated, but not label-free, since the node is selected with 500 ground-truth saliency masks."
However, we agree that explicitly mentioning the "small calibration set" helps clarify the scope of our work. We have appended "with a small calibration set" to "By simply identifying and selecting these nodes" in the Introduction, and in the Results part of Section 7 we have revised "Node-level selection performs best in nearly all settings, ..." to "Given a small calibration set, node-level selection performs best in nearly all settings, ...".