ViT Attention Weight as a Hierarchical Tree

While attention weights in Vision Transformers (ViTs) have become a central signal for tasks such as object localization, segmentation, and attribution, they still remain noisy and difficult to interpret. To address this, we analyze attention at the level of individual query-key pairs rather than aggregated attention weights. By defining the concatenation of query and key vectors as a *QK feature*, we first show that QK features exhibit structured patterns rather than forming a single homogeneous cluster. Based on this observation, we organize QK features into a hierarchical structure via recursive clustering, yielding a *QK tree* for each attention head. We verify that each node in the QK tree encodes a consistent semantic role, showing that distinct QK feature patterns capture distinct semantics. To interpret these roles, we introduce the *role card*, a diagnostic tool that visualizes the functional behavior of each node. Using this decomposition, we further identify *position-indicator nodes*, which show that spatial selectivity is localized within specific nodes rather than distributed across the entire head. Finally, we demonstrate that selecting a single *object-centric node* yields a simple and effective object localization method.