acceptodds
Under review as a conference paper at ICLR 2027

WHEN ATTENTION GEOMETRY SHIFTS: CALIBRATED STRUCTURAL EVALUATION OF VISION TRANSFORMERS

Abstract

Vision transformers route information through attention, yet shift detection often relies only on pooled features or classifier confidence. This paper studies whether a query’s routing pattern can be compared with nominal executions to characterize visual shift. Attention-Geometry Tracking (ATH) forms a calibration bank of clean embeddings and attention summaries, retrieves a coverage-qualified Local reference or falls back to an Aggregate reference, and scores shifts in key marginal, stable rank, entropy dispersion, and mean entropy. It requires neither labels nor classifier logits at scoring time and records the reference branch for every query. On a 75-condition window-averaged Swin-B corruption grid, mean-entropy shift (MES) is the strongest tested standalone score (mean AUROC 0.8049); the evaluated transferred-weight fusion reaches 0.6407. In a four-model OOD experiment—ViT-B/16, Swin-B, DINOv2-B/14, and CLIP ViT-B/16—Local references improve ImageNet-R AUROC over Aggregate references for every model, whereas DTD effects are mixed. Reference conditioning can therefore alter rankings but is shift dependent; high ImageNet-R FPR@95 precludes an operating- point claim. The corruption grids are explicitly exploratory because their selection slices overlap evaluation sources.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision transformers, out-of-distribution detection, attention geometry, training-free monitoring, self-supervised representations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DzzNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Attention-Geometry Tracking (ATH), a method that evaluates visual distribution shifts by analyzing a Vision Transformer's internal attention routing patterns rather than relying solely on pooled features or classifier confidence. ATH operates without labels or classifier logits at inference time by comparing query attention against a calibration reference bank of clean embeddings and reference statistics. Across a 75-condition Swin-B corruption grid, mean-entropy shift (MES) performed as the strongest standalone structural score. In four-model (ViT-B/16, Swin-B, DINOv2-B/14, and CLIP ViT-B/16) OOD evaluations, Local references consistently improved ImageNet-R AUROC over Aggregate references, although mixed results on DTD and high false-positive rates highlight that its utility is shift-dependent. Consequently, these findings do not establish a universally superior OOD detector, but rather a narrower mechanism result: exposing the reference branch, component scores, and fallback rate reveals when locality changes ranking - and when it does not.

Critical strengths

These are the paper's strengths:

  • it offers a novel insight on internal attention geometry: unlike traditional shift detection that relies solely on pooled features, class tokens, or classifier confidence logits, the paper explores whether a query's internal routing patterns across layers and heads can characterize visual shifts
  • the ATH framework operates without requiring ground-truth labels or classifier logits at scoring time, making it suitable for encoders that lack a task-specific classifier head
  • the approach is evaluated across diverse four-model OOD setup (ViT-B/16, Swin-B, DINOv2-B/14, and CLIP ViT-B/16)
  • the paper introduces individual metrics (RMD, SRD, ARD, and MES) in order to identify which components drive performance
  • repeating the Swin-B evaluation a second time with a new random seed (seed 43) helps check whether the main trends hold up across different data samples
  • the paper provides enough information like hyperparameters, reference bank sizes, retrieval counts and implementation details in order to facilitate results reproducibility
Critical weaknesses

These are the paper's weaknesses:

  • one of the most critical weaknesses is represented by the overall paper structure and presentation: a) placing "Related Work" in Section 3 (roughly in the middle of the paper, after detailing the core methodology and calibration equations) disrupts the standard academic flow, where background and related work typically precede technical formulation.
    b) the writing style seems more like an internal technical report than a polished research paper, lacking a clear, compelling narrative that clearly explains what the authors fundamentally want to achieve or prove. c) the introduction is very brief, failing to adequately motivate the core problem or clearly articulate the specific limitations of previous work that necessitate a new structural tracking approach.
    d) the related work section is too short and fails to cover recent relevant literature in the field, relying heavily on older citations. The newest cited references date back to 2022.
  • the false positive rate on ImageNet-R remained exceptionally high across all four models (84.7%–90.9%), indicating that the method is not reliable for practical settings
  • transferring scoring weights directly from ViT to Swin-B caused poor performance, highlighting a lack of robust hyperparameter tuning for different architectures
  • the paper lacks completed corruption grids for DINOv2 and CLIP, limiting the generalizability of the corruption findings beyond ViT and Swin.
  • ARD metric performed poorly on the Swin model (0.4556 AUROC). Because the data were processed in a way that lacks details, it is impossible to tell whether the entropy actually shrank or expanded.
  • evaluating OOD performance on only two datasets (ImageNet-R and DTD) limits the assessment of the method’s generalizability, as it remains unclear how well the approach performs under different distribution shifts, or on different data types
Questions for authors

These are my concerns:

  • the authors should address urgently the critical weakness represented by the paper structure and presentation mentioned above
  • given that the Local reference method results in exceptionally high false-positive rates (FPR@95 between 0.847 and 0.909) on ImageNet-R, what utility does this method offer for practical applications where low false alarms are critical?
  • why did the full ATH fusion configuration rely on weights transferred unchanged from ViT to Swin-B without proper adaptation? Can you provide proper cross-validation results for Swin-B?
  • why were corruption tests run on ViT and Swin but left out for DINOv2 and CLIP? Does MES work just as well across all four models when tested on the same setup?
  • to better evaluate the generalizability of your approach, could you provide the OOD performance on more challenging datasets such as Imagenet-A and Imagenet-O?
3 comments
AuthorsOpenReview ↗

Response: FPR@95, four-model corruption grids, and ImageNet-A/O results

Thank you for the detailed review. We agree that false alarms, incomplete corruption coverage and presentation require direct answers.

Practical utility and FPR@95. The ImageNet-R false-positive rates do not support reliable deployment at 95% shifted-image recall, and the new evidence does not remove this limitation. Across four models and two seeds, MES-Local FPR@95 is 78.2%–90.0% on ImageNet-R, 84.4%–94.6% on ImageNet-A and 71.0%–89.6% on ImageNet-O. These values concern MES, whereas the submitted figures concerned full ATH fusion. Convention: shifted images are the positive class, and FPR@95 is the fraction of clean controls flagged at the first threshold reaching at least 95% shifted-image recall. Cross-paper comparisons must also match the control distribution, model, calibration size and dataset sampling; the same metric name alone does not make protocols comparable. DTD is substantially easier: AUROC 0.9274–0.9664, FPR@95 16.2%–33.2%, and MES beats the fused score in all eight runs. Even so, the evidence supports structural-shift analysis and comparison of reference choices, not a validated low-false-alarm system. At empirical FPR ≤5%, MES-Local detects 9.7–13.0% of ImageNet-A and 16.0–32.5% of ImageNet-O examples. These are evaluation ROC points, not independently calibrated thresholds; that validation remains pending.

Transferred weights and validation. The submitted Swin experiment transferred ViT-derived weights. The new experiments derive weights separately for each model/seed from reserved selection images, then evaluate on disjoint images. Model-specific selection does not remove the observed fusion deficit: Swin mean AUROC over the 75 conditions is 0.6590/0.6709 for full ATH versus 0.7978/0.7996 for MES (seeds 42/43). These are two seed/split sensitivity runs, not k-fold cross-validation, and I do not claim the selected weights are optimal. The negative fusion result will be reported explicitly.

Complete corruption coverage. All four models now have a completed seed-42 grid of 15 corruption types × 5 severities. MES/fusion mean AUROC is 0.7444/0.7132 (ViT), 0.7978/0.6590 (Swin), 0.8239/0.7005 (DINOv2) and 0.7707/0.6598 (CLIP); ViT and Swin also have seed-43 grids. MES exceeds fusion on every grid and is the strongest standalone component on all four backbones, though its absolute level varies by architecture. These grids use disjoint selection/evaluation images (corruption families are shared) and do not retroactively repair the submitted exploratory results.

ARD direction. On the new Swin grids, ARD alone yields 0.4510/0.4707 AUROC, and removing ARD raises fusion to 0.7759/0.7719. This ablation favors removing ARD, but does not establish whether entropy dispersion contracted or expanded: the rectified score cannot recover that sign. A signed query-versus-reference IQR analysis is pending.

ImageNet-A/O. Both are now evaluated for all four backbones and both seeds. MES-Local AUROC is 0.5630–0.6380 on ImageNet-A and 0.6711–0.7782 on ImageNet-O. Feature methods are stronger on several backbones. For example, ViM FPR@95 on ImageNet-A is 50.4%/53.4% (ViT) and 44.4%/44.6% (Swin), against 91.6%/93.0% and 84.8%/84.4% for MES-Local. For MES, the Local-over-Aggregate AUROC gain on A/O is significant only for Swin at both seeds, under the separate 176-comparison A/O Holm family. ImageNet-A is treated as a hard in-category shift, separately from ImageNet-O's semantic OOD; the controls are ImageNetV2, not the canonical benchmark protocol. Novelty to each pretrained backbone is unestablished: broader pretraining may include these categories.

Presentation. The revision will expand the motivation, place related work with recent literature before the method, and move implementation details to the appendix. An early pipeline figure and cross-model tables will clarify the evidence. These manuscript changes are pending.

Reviewer_DzzNOpenReview ↗

Response to Authors

I want to thank the authors for addressing some of my concerns and running the extra experiments on weights transfer, across all four models and additional OOD performance on ImageNet-A/O datasets.However, these new numbers confirm that the full ATH fusion doesn't outperform simpler baselines, suffers from very high false-positive rates, and struggles on more challenging datasets (ImageNet-AO). Besides these limitations, the paper still requires major structural and narrative revisions. For all these conisderations, I do not consider the paper is mature enough to be considered for publication, and therefore I maintain my initial rating.

AuthorsOpenReview ↗

Follow-up to Reviewer DzzN on empirical findings and paper revisions

Thank you for your follow-up and for assessing the additional experiments. We have uploaded the revised paper. We agree that the evidence does not show a general advantage for ATH fusion or reliable high-recall detection across these shifts.

The revision aligns the claims with this evidence: it presents MES as the primary score, treats fusion as an ablation, and states the operating limits directly.

Fusion: MES is now the primary score and fusion is an ablation; MES exceeds model-specific fusion on all six corruption grids (Section 5.4; design limitations in Appendix B).

False alarms and harder shifts: score overlap and operating limits are reported directly (Section 5.5, Figure 3). Full ViM significantly exceeds MES on ImageNet-A for all ViT/Swin runs (Section 5.3).

Structure: the problem setting now precedes the method; same-example baselines, including reference-free attention entropy, are in the main text (Section 5.3, Table 3); and limitations are consolidated in Section 6.

We recognize that these revisions do not remove the limitations you identified; they clarify which conclusions the evidence supports.

Reviewer_TyYtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ATH evaluates visual distribution shifts by comparing attention statistics with embedding-conditioned references from a clean calibration bank, using an aggregate fallback when reference coverage is low.

Critical strengths
  1. The paper addresses a highly relevant problem, and its scoring requires neither labels nor classifier logits at inference time.

  2. The held-out OOD experiments cover four different backbones, and the authors transparently report negative findings and evaluation limitations.

Critical weaknesses
  1. Limited novelty. The main contribution is reference conditioning of established attention statistics, with limited methodological advancement and insufficient evidence of benefits beyond simpler alternatives.

  2. Poorly written. Excessive configuration detail makes the narrative difficult to follow, and the scoring variant used in Table 2 is insufficiently specified.

  3. Incomplete independent validation. The corruption and feature-baseline evaluations overlap with parameter-selection sources, leaving these findings without fully independent confirmation.

  4. Unsubstantiated fusion benefits. The evaluated fusion underperforms MES on both backbones; Swin AUROC is 0.6407 versus 0.8049 (Table 1), leaving the additional components insufficiently justified.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response: disjoint-split re-evaluation, fusion result, and reference controls

Thank you for identifying the evaluation overlap and questioning fusion. We agree that the submitted corruption evidence lacked independent confirmation and that the score definitions were unclear.

Independent evaluation. After submission we ran new evaluations under a corrected protocol. Within each model/seed run, image sources are split into disjoint partitions: 500 calibration, 1,000 selection, and 500 evaluation images (zero overlap). The corrected protocol fixes the retrieval count at k = 10 before these evaluation runs. The coverage threshold and model-specific fusion weights are derived from selection images only and frozen before evaluation. Feature baselines (KNN-OOD, Mahalanobis, residual) are fit on the calibration partition only. One limitation remains: the weights are derived on the same 15 corruption families that are evaluated, so images are disjoint but corruption families are not. The table reports mean AUROC over the 75 corruption-severity conditions:

BackboneMES-Local, seed 42ATH-Local fusion, seed 42
ViT-B/160.74440.7132
Swin-B0.79780.6590
DINOv2-B/140.82390.7005
CLIP ViT-B/160.77070.6598

At seed 43, MES/fusion are 0.7453/0.6983 (ViT) and 0.7996/0.6709 (Swin). These new runs do not retroactively validate the submitted grids. Full-grid means are descriptive; paired source-blocked inference covers eight conditions only.

Fusion and simpler alternatives. Your fusion concern stands: The submitted Table 1 fusion (Swin 0.6407 vs 0.8049 for MES) used ViT-derived weights transferred unchanged; the new runs instead derive weights separately for each model and seed, and fusion still trails MES (Swin 0.6590 vs 0.7978 at seed 42). Model-specific weights do not produce a fusion benefit in the completed corruption grids. MES exceeds the fused score in every completed grid; in the eight-condition subset the difference is not significant for ViT. The revised paper will therefore not claim that four-component fusion is beneficial. On simpler alternatives, the corruption evidence is favorable to MES. In the eight-condition subset, MES-Local exceeds KNN-OOD (+0.16 to +0.49 AUROC), class-agnostic Mahalanobis and residual-feature scores (+0.10 to +0.50), and their normalized-feature variants (+0.14 to +0.42) on all four backbones, with every simultaneous interval excluding zero within each model's declared comparison family. The residual score is only the feature term, not full ViM. On the full ViT and Swin grids it also exceeds MSP, Energy, DOCTOR and logit entropy (descriptively; the smallest mean-AUROC margin is 0.0132, Swin seed 43). OOD results are less favorable. On ImageNet-R, ViM outperforms MES-Local on ViT and Swin, and raw Mahalanobis does so on CLIP, at both seeds (Holm-significant). On DTD, aggregate references are already strong and plain attention entropy is competitive on DINOv2.

Contribution and score definition. The statistics and retrieval are established tools. The question is whether embedding-conditioned attention references improve shift detection, and when aggregate references are preferable. MES-Local scores deviation of per-head mean attention entropy from that reference; ATH-Local fuses four component deviations. In both Local variants, insufficient similarity triggers the aggregate reference. The submitted OOD table used fused ATH, not MES; the new comparisons distinguish the two explicitly. On ImageNet-R, MES-Local improves AUROC over MES-Aggregate in all eight model/seed runs (+0.026 to +0.087, all significant in the 176-comparison Holm family). On the newly added ImageNet-A/O, that locality effect is significant only for Swin. This supports a shift-dependent reference effect, not general detector superiority.

Completed reference controls. Reusing cached embeddings and attention summaries, we held the original coverage decision and nearest-neighbor softmax weights fixed while replacing nearest anchors with uniformly sampled random anchors. On ImageNet-R, nearest anchors improve AUROC in all eight runs (+0.033 to +0.101; seven significant in a separate 128-test Holm family). No difference survives correction on DTD; two of eight favor nearest anchors on each of ImageNet-A/O. Keeping fallback versus always-local has mixed effects on ImageNet-R: −0.018 to +0.022 AUROC, with a significant benefit for Swin seed 43 and harm for CLIP seed 42. These post-hoc controls use one fixed randomization and support a shift-dependent role for relevant anchors, not universally beneficial fallback or low false-alarm deployment.

Presentation. The revision will introduce the problem first, label scores consistently, and move configuration details to an appendix.

Reviewer_zAq7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues for attenton geometry tracking. Basically the key idea is to build a calibration bank of clean embeddings + attention based per-anchor statistics. The system `switches’ between local reference/ mean reference based on a particular threshold decision.

Critical strengths

The core premise of the paper is interesting, that distribution shifts can change the internal attention patterns in the model

Critical weaknesses

To be honest, i had a lot of issues in reading the paper.

For example,

  • There is no figure to look at other than figure 1, at page 5.
  • section 2 directly starts talking about calibration anchors etc, and it is very difficult to understand what exact problem is being solved. perhaps a dedicated problem setup section can be helpful
  • Table 2 compares only Local vs Aggregate. There is no MSP, Energy, KNN or ViM on the same data, only two shifts, and no standard OOD benchmarks.
    • A lot of components are standard and the retrieval resembles KNN-OOD.
Questions for authors

Questions

  • What is the core problem here which is trying to be solved?
1 comment
AuthorsOpenReview ↗

Response: problem setup, KNN-OOD distinction, and same-example baselines

Thank you for identifying the unclear problem setup and missing same-example baselines. We agree that the equations obscured the question.

What problem is being solved? Given a frozen vision encoder and a clean calibration collection, the task is to score whether a new image produces attention statistics that depart from the clean reference. It is image-level shift detection, not a guarantee of classification error or semantic novelty. Scoring requires neither labels nor classifier logits, but requires internal attention access. This includes the evaluated DINOv2 encoder without a task-specific classifier. Corruption, rendition, texture and semantic shifts are distinct tasks and are evaluated separately.

How does this differ from KNN-OOD? KNN-OOD scores an image by its embedding distance to clean neighbors. Here, embedding similarities select and weight clean attention statistics to construct a reference; deviation from that reference, rather than neighbor distance alone, is the MES score. Below a fixed similarity threshold, the aggregate clean reference is used. Empirically, on the same examples:

  • DTD: MES-Local exceeds KNN-OOD in all eight model/seed runs (+0.07 to +0.55 AUROC; Holm-significant).
  • Corruption subset: MES-Local exceeds KNN-OOD on all four backbones (+0.16 to +0.49), with simultaneous intervals excluding zero within each model's comparison family. This covers eight conditions, not all 75.
  • ImageNet-R: mixed. MES-Local is better on DINOv2 (both seeds) and ViT seed 43, worse on Swin (both seeds), and not statistically distinguishable in the other comparisons (Holm-adjusted); this is not evidence of equivalence.

New same-example comparisons. All four models use seeds 42/43, 500 controls and 1,000 images per shift. Same-example methods: KNN-OOD, raw/normalized Mahalanobis and residual scores, plain attention entropy, plus MSP, Energy, logit entropy and ViM for the supervised ViT/Swin classifiers. ImageNet-R AUROC, seed 42 / seed 43:

BackboneMES-LocalMES-AggregateNamed comparator
ViT-B/160.6127 / 0.61610.5779 / 0.5901ViM: 0.8204 / 0.8156
Swin-B0.7425 / 0.72510.6691 / 0.6478ViM: 0.8100 / 0.8107
DINOv2-B/140.6835 / 0.65750.6269 / 0.6266Plain attention entropy: 0.6951 / 0.7054*
CLIP ViT-B/160.6740 / 0.67750.5866 / 0.6070Raw Mahalanobis: 0.7921 / 0.8008

*Significantly above MES-Local at seed 43 only under Holm correction. R/DTD AUROC tests share one 176-comparison family; A/O use a separate 176-comparison family. Seeds are not pooled. These are named counterexamples; the appendix will report all methods. ViM uses the 500-image calibration set and principal dimension 128, rather than the original large-calibration configuration. Logit methods are inapplicable to the evaluated DINOv2/CLIP vision-only setup.

On DTD, aggregate MES is already strong (AUROC 0.9464–0.9653); locality is not universally necessary.

Additional challenging datasets. We have added ImageNet-O (semantic OOD relative to ImageNet-1K) and ImageNet-A (a challenging in-category shift), using the same protocol with ImageNetV2 controls, not the canonical benchmark. Novelty to each pretrained backbone is unestablished: broader pretraining may include these categories. MES-Local reaches AUROC 0.6711–0.7782 on ImageNet-O and 0.5630–0.6380 on ImageNet-A. On ImageNet-A, ViM is far stronger on ViT and Swin (AUROC 0.8307–0.8442). On ImageNet-O, only the ViT seed-42 ViM–MES AUROC difference survives Holm correction. ViM has lower observed FPR@95 on both datasets for ViT/Swin at both seeds. I report ImageNet-A as a clear limitation.

Completed reference controls. Reusing cached embeddings and attention summaries, we held the original coverage decision and nearest-neighbor softmax weights fixed while replacing nearest anchors with uniformly sampled random anchors. On ImageNet-R, nearest anchors improve AUROC in all eight runs (+0.033 to +0.101; seven significant in a separate 128-test Holm family). No difference survives correction on DTD; two of eight favor nearest anchors on each of ImageNet-A/O. Keeping fallback versus always-local has mixed effects on ImageNet-R: −0.018 to +0.022 AUROC, with a significant benefit for Swin seed 43 and harm for CLIP seed 42. These post-hoc controls use one fixed randomization and support a shift-dependent role for relevant anchors, not universally beneficial fallback or low false-alarm deployment.

Presentation. The revision will add problem setup, an early pipeline figure and a four-model table, moving configuration details to the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Summary of revisions

We thank all reviewers for their feedback. We have uploaded a revised paper. The main changes:

Claims aligned with the evidence: MES is the primary evaluated score and ATH fusion is reported as an ablation (Section 5.4). The contribution is stated as an evaluation of when clean-reference conditioning helps attention-based shift detection under a shared 500-image clean bank.

Leakage-free protocol: Corruption results are re-run with calibration, selection and evaluation images disjoint within each run (Section 4, Appendix A). They replace the submission’s selection-overlapping results.

Expanded comparisons: Same-example comparisons now include classifier scores, full ViM and reference-free attention entropy (Sections 5.1 and 5.3; Tables 2 and 3). Matched controls test nearest versus random references (Section 5.2).

Operating limits: Score overlap and recall at a clean-calibrated 5% false-alarm budget are reported directly (Section 5.5, Figure 3).

Structure and scope: The problem setting precedes the method. Attention-based OOD related work is added, and shift monitoring is distinguished from full-spectrum OOD detection (Section 2). Limitations are consolidated in Section 6.

We appreciate the reviewers’ time and the improvements their comments prompted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.