- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Attention-Geometry Tracking (ATH), a method that evaluates visual distribution shifts by analyzing a Vision Transformer's internal attention routing patterns rather than relying solely on pooled features or classifier confidence. ATH operates without labels or classifier logits at inference time by comparing query attention against a calibration reference bank of clean embeddings and reference statistics. Across a 75-condition Swin-B corruption grid, mean-entropy shift (MES) performed as the strongest standalone structural score. In four-model (ViT-B/16, Swin-B, DINOv2-B/14, and CLIP ViT-B/16) OOD evaluations, Local references consistently improved ImageNet-R AUROC over Aggregate references, although mixed results on DTD and high false-positive rates highlight that its utility is shift-dependent. Consequently, these findings do not establish a universally superior OOD detector, but rather a narrower mechanism result: exposing the reference branch, component scores, and fallback rate reveals when locality changes ranking - and when it does not.
Critical strengths
These are the paper's strengths:
- it offers a novel insight on internal attention geometry: unlike traditional shift detection that relies solely on pooled features, class tokens, or classifier confidence logits, the paper explores whether a query's internal routing patterns across layers and heads can characterize visual shifts
- the ATH framework operates without requiring ground-truth labels or classifier logits at scoring time, making it suitable for encoders that lack a task-specific classifier head
- the approach is evaluated across diverse four-model OOD setup (ViT-B/16, Swin-B, DINOv2-B/14, and CLIP ViT-B/16)
- the paper introduces individual metrics (RMD, SRD, ARD, and MES) in order to identify which components drive performance
- repeating the Swin-B evaluation a second time with a new random seed (seed 43) helps check whether the main trends hold up across different data samples
- the paper provides enough information like hyperparameters, reference bank sizes, retrieval counts and implementation details in order to facilitate results reproducibility
Critical weaknesses
These are the paper's weaknesses:
- one of the most critical weaknesses is represented by the overall paper structure and presentation:
a) placing "Related Work" in Section 3 (roughly in the middle of the paper, after detailing the core methodology and calibration equations) disrupts the standard academic flow, where background and related work typically precede technical formulation.
b) the writing style seems more like an internal technical report than a polished research paper, lacking a clear, compelling narrative that clearly explains what the authors fundamentally want to achieve or prove. c) the introduction is very brief, failing to adequately motivate the core problem or clearly articulate the specific limitations of previous work that necessitate a new structural tracking approach.
d) the related work section is too short and fails to cover recent relevant literature in the field, relying heavily on older citations. The newest cited references date back to 2022. - the false positive rate on ImageNet-R remained exceptionally high across all four models (84.7%–90.9%), indicating that the method is not reliable for practical settings
- transferring scoring weights directly from ViT to Swin-B caused poor performance, highlighting a lack of robust hyperparameter tuning for different architectures
- the paper lacks completed corruption grids for DINOv2 and CLIP, limiting the generalizability of the corruption findings beyond ViT and Swin.
- ARD metric performed poorly on the Swin model (0.4556 AUROC). Because the data were processed in a way that lacks details, it is impossible to tell whether the entropy actually shrank or expanded.
- evaluating OOD performance on only two datasets (ImageNet-R and DTD) limits the assessment of the method’s generalizability, as it remains unclear how well the approach performs under different distribution shifts, or on different data types
Questions for authors
These are my concerns:
- the authors should address urgently the critical weakness represented by the paper structure and presentation mentioned above
- given that the Local reference method results in exceptionally high false-positive rates (FPR@95 between 0.847 and 0.909) on ImageNet-R, what utility does this method offer for practical applications where low false alarms are critical?
- why did the full ATH fusion configuration rely on weights transferred unchanged from ViT to Swin-B without proper adaptation? Can you provide proper cross-validation results for Swin-B?
- why were corruption tests run on ViT and Swin but left out for DINOv2 and CLIP? Does MES work just as well across all four models when tested on the same setup?
- to better evaluate the generalizability of your approach, could you provide the OOD performance on more challenging datasets such as Imagenet-A and Imagenet-O?
3 comments
Response to Authors
I want to thank the authors for addressing some of my concerns and running the extra experiments on weights transfer, across all four models and additional OOD performance on ImageNet-A/O datasets.However, these new numbers confirm that the full ATH fusion doesn't outperform simpler baselines, suffers from very high false-positive rates, and struggles on more challenging datasets (ImageNet-AO). Besides these limitations, the paper still requires major structural and narrative revisions. For all these conisderations, I do not consider the paper is mature enough to be considered for publication, and therefore I maintain my initial rating.
Follow-up to Reviewer DzzN on empirical findings and paper revisions
Thank you for your follow-up and for assessing the additional experiments. We have uploaded the revised paper. We agree that the evidence does not show a general advantage for ATH fusion or reliable high-recall detection across these shifts.
The revision aligns the claims with this evidence: it presents MES as the primary score, treats fusion as an ablation, and states the operating limits directly.
Fusion: MES is now the primary score and fusion is an ablation; MES exceeds model-specific fusion on all six corruption grids (Section 5.4; design limitations in Appendix B).
False alarms and harder shifts: score overlap and operating limits are reported directly (Section 5.5, Figure 3). Full ViM significantly exceeds MES on ImageNet-A for all ViT/Swin runs (Section 5.3).
Structure: the problem setting now precedes the method; same-example baselines, including reference-free attention entropy, are in the main text (Section 5.3, Table 3); and limitations are consolidated in Section 6.
We recognize that these revisions do not remove the limitations you identified; they clarify which conclusions the evidence supports.
Response: FPR@95, four-model corruption grids, and ImageNet-A/O results
Thank you for the detailed review. We agree that false alarms, incomplete corruption coverage and presentation require direct answers.
Practical utility and FPR@95. The ImageNet-R false-positive rates do not support reliable deployment at 95% shifted-image recall, and the new evidence does not remove this limitation. Across four models and two seeds, MES-Local FPR@95 is 78.2%–90.0% on ImageNet-R, 84.4%–94.6% on ImageNet-A and 71.0%–89.6% on ImageNet-O. These values concern MES, whereas the submitted figures concerned full ATH fusion. Convention: shifted images are the positive class, and FPR@95 is the fraction of clean controls flagged at the first threshold reaching at least 95% shifted-image recall. Cross-paper comparisons must also match the control distribution, model, calibration size and dataset sampling; the same metric name alone does not make protocols comparable. DTD is substantially easier: AUROC 0.9274–0.9664, FPR@95 16.2%–33.2%, and MES beats the fused score in all eight runs. Even so, the evidence supports structural-shift analysis and comparison of reference choices, not a validated low-false-alarm system. At empirical FPR ≤5%, MES-Local detects 9.7–13.0% of ImageNet-A and 16.0–32.5% of ImageNet-O examples. These are evaluation ROC points, not independently calibrated thresholds; that validation remains pending.
Transferred weights and validation. The submitted Swin experiment transferred ViT-derived weights. The new experiments derive weights separately for each model/seed from reserved selection images, then evaluate on disjoint images. Model-specific selection does not remove the observed fusion deficit: Swin mean AUROC over the 75 conditions is 0.6590/0.6709 for full ATH versus 0.7978/0.7996 for MES (seeds 42/43). These are two seed/split sensitivity runs, not k-fold cross-validation, and I do not claim the selected weights are optimal. The negative fusion result will be reported explicitly.
Complete corruption coverage. All four models now have a completed seed-42 grid of 15 corruption types × 5 severities. MES/fusion mean AUROC is 0.7444/0.7132 (ViT), 0.7978/0.6590 (Swin), 0.8239/0.7005 (DINOv2) and 0.7707/0.6598 (CLIP); ViT and Swin also have seed-43 grids. MES exceeds fusion on every grid and is the strongest standalone component on all four backbones, though its absolute level varies by architecture. These grids use disjoint selection/evaluation images (corruption families are shared) and do not retroactively repair the submitted exploratory results.
ARD direction. On the new Swin grids, ARD alone yields 0.4510/0.4707 AUROC, and removing ARD raises fusion to 0.7759/0.7719. This ablation favors removing ARD, but does not establish whether entropy dispersion contracted or expanded: the rectified score cannot recover that sign. A signed query-versus-reference IQR analysis is pending.
ImageNet-A/O. Both are now evaluated for all four backbones and both seeds. MES-Local AUROC is 0.5630–0.6380 on ImageNet-A and 0.6711–0.7782 on ImageNet-O. Feature methods are stronger on several backbones. For example, ViM FPR@95 on ImageNet-A is 50.4%/53.4% (ViT) and 44.4%/44.6% (Swin), against 91.6%/93.0% and 84.8%/84.4% for MES-Local. For MES, the Local-over-Aggregate AUROC gain on A/O is significant only for Swin at both seeds, under the separate 176-comparison A/O Holm family. ImageNet-A is treated as a hard in-category shift, separately from ImageNet-O's semantic OOD; the controls are ImageNetV2, not the canonical benchmark protocol. Novelty to each pretrained backbone is unestablished: broader pretraining may include these categories.
Presentation. The revision will expand the motivation, place related work with recent literature before the method, and move implementation details to the appendix. An early pipeline figure and cross-model tables will clarify the evidence. These manuscript changes are pending.