acceptodds
Under review as a conference paper at ICLR 2027

Reconstruction, Not Localization: A Mechanistic Account of Sparse-Feature OOD Detection in Vision Transformers

Abstract

Sparse autoencoder (SAE) features have been proposed as an interpretable basis for out-of-distribution (OOD) detection, inviting the reading that SAEs contain localized "OOD units". We test the controls such a claim needs on frozen vision transformers (13 OOD sets, CIFAR-10-C, a 100-class in-distribution (ID) dataset, and ImageNet-1K scale). In both ReLU and exact-k TopK SAEs (three seeds each), frequently firing units are seed-idiosyncratic, fixed sets of up to five units add about one point of confidence collapse on ImageNet-O beyond the plain reconstruction round-trip, and the largest-contribution units are in-distribution anchors. The collapse is instead the SAE encode->decode round-trip: it lowers a frozen ImageNet head's confidence on ImageNet-O by 70% versus 11% on ID (47% with its native CLS input), whereas matched noise moves it by only 7%; a PCA round-trip reproduces the collapse (73%). It is head-dependent: at a fixed ID, the drop grows monotonically with the number of head classes (from -3% at 10 to 70% at 1,000), while heads trained on the ID label space stay inert (4.5%). We call this Mismatched Wide-Head Reconstruction Sensitivity; it persists with heads trained at each width and on a second backbone (DeiT-B/16). Against published baselines, SAE-based detection is competitive but not better, and a PCA basis matches it. We release a six-step Sparse Reconstruction Audit with margin-reported decision criteria, all JSON artifacts, and a runnable demo; on a released third-party SAE trained on ImageNet it finds no such effect, and a controlled comparison at the same hook ties the effect to an SAE fit on the audited ID. The contribution is not a stronger detector but a tested boundary on what sparse-feature OOD analysis can support.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, out-of-distribution detection, mechanistic interpretability, vision transformers, reconstruction, feature stability, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eVUdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the use of SAEs on OOD detection problem. Authors start with a negative result on SAE OOD units, linking it to SAEs nonidentifiability across different seeds. Then they make the observation that SAEs' usefulness for OOD detection comes mainly from reconstruction gap, rather than specific OOD units. SAE encode-decode is a lossy transformation that mainly results in a confidence drop on the linear probe estimator. Overall, the paper presents a systematic study to investigate these effects.

Critical strengths
  • Given the existing works' claims, demonstrating a negative result is valuable. Stressing unidentifability of SAEs challenges the existence of specific OOD units for detection problem.
  • Understanding the mechanism of why SAE work for OOD by checking its reconstruction effects is insightful.
Critical weaknesses
  • Paper uses a lot of terminology, is very dense, repetitive and hard to follow. Authors explicitly acknowledge substantial AI assistance for drafting but I personally do not think this is at a publishable level.
  • Evaluation protocol is far form the modern OOD detection evaluations. There is no structured near and far OOD analysis and the used benchmark seems very outdated and at the saturation level.
  • Non-identifiability of SAEs and instability of its features are already well-known. Cited works with SAE OOD units are very recent preprints which do not show strong performance across architectures and datasets. Together, this partially challenges the motivation.
  • Conclusions are underwhelming. Methodology is not well tested across benchmarks. Even then, this acts more as an observational result rather than a solution to an existing problem. Which itself could be valuable when the results are counter intuitive or well tested. I think the work provides a useful preliminary study but its conceptual insights seem limited for a conference paper.
Questions for authors
  • I am curious about the performance of SAE as a detector on more recent standardized benchmarks like OpenOOD. Could the authors run additional experiments across different architectures to check if reconstruction effects are consistent?
2 comments
AuthorsOpenReview ↗

Part 1/2: OpenOOD v1.5 results and the round-trip drop across backbones

Thank you for the review. Your question asks for exactly the evaluation the submission lacked, so we ran it.

In short: SAE reconstruction is not a strong detector on OpenOOD v1.5, and the round-trip confidence drop has the same direction on the three supervised backbones we ran, with a size that depends on the backbone and the SAE.

Detection. OpenOOD v1.5 CIFAR-10 benchmark (OpenOOD's image lists, near/far groups and metric code; every image resized to 32 px on its shorter side and center-cropped to 32×32 px, then our 224-px input transform); ViT-B/16, mean pooling, the paper's CIFAR-10 SAE. MSP, Energy and ViM use a CIFAR-10 linear probe on the frozen features; MSP and Energy use it with temperature scaling fit on in-distribution (ID) validation data, and in brackets without it (T = 1, as in OpenOOD). AUROC / FPR@95. We keep frozen ImageNet backbones instead of OpenOOD's training protocol, so these numbers are not comparable with its leaderboard (our reply to 8Wfp lists every difference).

MethodNear-OODFar-OOD
MSP0.908 / 0.302 (0.892 / 0.319)0.914 / 0.260 (0.897 / 0.281)
Energy0.925 / 0.292 (0.925 / 0.287)0.944 / 0.196 (0.943 / 0.200)
KNN0.867 / 0.5000.871 / 0.374
Mahalanobis0.860 / 0.5450.904 / 0.347
ViM0.953 / 0.2010.976 / 0.097
PCA reconstruction0.856 / 0.5690.904 / 0.358
SAE reconstruction0.847 / 0.5500.895 / 0.356
SAE activation Mahalanobis0.816 / 0.5770.879 / 0.375

Across seven configurations on three backbones (ViT-B/16 and DeiT-B/16 with mean and CLS pooling, ResNet-50, and ViT-B/16 mean and CLS with SAEs trained on all 50k images), ViM is best on near-OOD in all seven and on far-OOD in six (Energy on DeiT-B/16 CLS far-OOD). SAE reconstruction ranks 5th to 8th of 8 (4th to 8th at T = 1). It is below MSP on near-OOD in all seven configurations with the calibrated probe and in six at T = 1 (DeiT-B/16 mean: 0.895 against 0.887), and both SAE scores are below Energy in all 14 CIFAR-10 cells. On ImageNet-1K (ViT-B/16, 45,000 ID test images; mean-pooled features with the native head and a Resize(224) crop, which differ from OpenOOD's protocol), SAE reconstruction with three TopK SAE seeds reaches 0.706-0.710 AUROC on near-OOD and 0.906-0.907 on far-OOD, behind Mahalanobis (0.741), MSP (0.734) and ViM (0.721) on near-OOD and ViM (0.928) and Mahalanobis (0.923) on far-OOD.

Round-trip drop (a main-text table in the revision). Relative drop in the mean MSP of the frozen 1,000-class ImageNet head when each embedding is replaced by its SAE reconstruction, on the same OpenOOD sets (ID / mean over the near-OOD sets / mean over the far-OOD sets). Gaussian noise with the same squared error moves MSP by at most 2.1% on any of these means. Each row uses one SAE seed.

Backbone and SAEIDNearFar
ViT-B/16 mean, CIFAR-10 SAE10.7%50.2%43.4%
ViT-B/16 CLS, CIFAR-10 SAE11.5%55.1%39.3%
DeiT-B/16 mean, CIFAR-10 SAE24.9%82.6%79.1%
DeiT-B/16 CLS, CIFAR-10 SAE22.0%73.5%64.9%
ResNet-50, CIFAR-10 SAE2.0%9.5%16.8%
ViT-B/16 mean, SAE on all 50k CIFAR-10 images3.4%35.3%19.5%
ViT-B/16 CLS, SAE on all 50k CIFAR-10 images5.3%42.7%26.0%
ViT-B/16 mean, ImageNet-1K as ID, TopK SAE fit on ImageNet (seed 42)1.9%6.2%7.1%

The OOD drop is 2.9 to 10.3 times the ID drop on the CIFAR-10 benchmark (3.3 and 3.8 times in the ImageNet-1K row). This ratio reaches 5, the threshold we fixed before this run, in only 4 of the 14 CIFAR-10 cells (ResNet-50 far-OOD and three cells with the all-50k SAEs), on both groups of a configuration, as the check was written, in 1 of 7 (ViT-B/16 mean, all-50k SAE), and in none of the ImageNet-1K cells; when the audit also requires a drop of at least 20 points, in 2 of 14 cells. The drop is small at ImageNet-1K (1.9% here on OpenOOD's 45,000 ID images; 1.8% on the 50,000 validation images in our reply to 8Wfp), where the SAE is fit on the head's own domain and has more capacity (see our reply to 8Wfp, part 3).

One clarification: the large drops are on the wide ImageNet head. A CIFAR-10 linear probe, matched to the ID label space, is much less affected (between -3% and 14% on the near and far means).

Reference: Zhang et al. (2024), OpenOOD v1.5, DMLR (arXiv:2306.09301).

AuthorsOpenReview ↗

Part 2/2: Protocol, motivation, conclusions and writing

Evaluation protocol (W2). You were right that our benchmark was saturated, and the OpenOOD run showed one reason. CIFAR-10 images are 32 px, but several of our OOD sets were used at full resolution, which gives feature-space detectors a resolution cue. On OpenOOD's Textures and Places365 lists at stored resolution, SAE reconstruction reaches 1.000 and 0.995 AUROC (PCA and KNN behave the same way; MSP does not); with OpenOOD's 32-px step these fall to 0.961 and 0.834. The revision replaces Table 1 with the OpenOOD near/far table. The round-trip result survives the same correction: with ImageNet-O brought to 32 px, the drop is 62% versus 11% on ID (66% with CLS input), against 2% for matched noise; with CLS input and SAEs trained on all 50,000 images it is 54% versus 5%.

Motivation (W3). We agree that SAE non-identifiability is known (Paulo and Belrose, 2026; Fel et al., 2025) and that the SAE-OOD works we cite are recent (two of them preprints); none of them claims seed-stable OOD units (what each reports: our reply to 8Wfp, part 4). The motivation does not rest on them: splicing SAE reconstructions into a model and measuring the change in its loss is a standard SAE evaluation (Gao et al., 2025), and the round-trip drop is that evaluation read through a classifier's confidence. The revision cuts the unit-level part to one stability table and the result that five fixed units detect ImageNet-O within one SAE, moves the per-unit analyses to an appendix (Section 6 keeps a short summary), and presents it as confirming known instability in this pipeline. The finding we consider new is about what the confidence signal measures. In our experiments the round-trip drop concentrates on OOD inputs when the reconstruction is fit on a narrow ID and read by a head whose label space is much wider, and its size varies with the backbone and with the SAE: large for ViT-B/16 and DeiT-B/16 on CIFAR-10 (39-83% on the OpenOOD means with the paper's SAEs (trained on a 20,000-image subset), 20-43% with SAEs trained on all 50k images), smaller for ResNet-50 (10-17%) and with ImageNet-100 as ID (about 10% on ImageNet-O, under trained heads), and small at ImageNet-1K (6-7%). PCA-256 fit on the same ID training set gives a similar drop with the paper's ViT-B/16 SAE (73% against 70% on ImageNet-O) but not everywhere: 5% against the SAE's 14% on ResNet-50, 54% against 6% at ImageNet-1K, and 73% against 53% with the all-50k SAEs. Reading such drops as evidence about SAE features therefore needs these controls, and the audit provides them.

Conclusions (W4). We agree that the result is observational and diagnostic rather than a solution, and the new conclusion says so plainly, with three takeaways: SAE scores trail standard post-hoc detectors on OpenOOD; the units that detect OOD inputs are mostly specific to one training seed; and a round-trip confidence drop should be read only beside the noise, PCA and matched-head controls, because its size depends on the head and on the data the SAE is fit on. It is now tested on the OpenOOD CIFAR-10 and ImageNet-1K benchmarks, in addition to the original sets, and on three backbones (two ViTs and a CNN).

Writing (W1). We accept this. The revision rewrites the abstract, introduction and setup, and cuts repetition in the results sections and the conclusion. The task and the confidence measure are defined before any result, terms such as "seed-idiosyncratic" are removed or defined, the long name of the effect is dropped, and the audit is given as an algorithm box, with the equation of each score in Section 3.

References: Paulo and Belrose (2026), ICLR; Fel et al. (2025), ICML; Gao et al. (2025), ICLR.

Reviewer_8WfpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

General Comment: The presentation of the paper is bad. It looks very unpolished and gives the impression of being generated almost entirely by an LLM. This paper is concerning because it shows many signals of AI-generated slop: contradictory statements between em dashes (“—”), references to files that are not included in the paper, numbers without explanations, among many other issues. The introduction and abstract are really hard to read, and it is difficult to understand the point of this paper. The paper has disconnected paragraphs, none of which properly explain why the analysis is necessary or provide a natural flow. It is just filled with findings without any real insight, often simply rewriting the results and metrics or presenting numbers that are not very well explained.

Below a more detailed description about the different sections:

Abstract The authors state that SAE features invite the interpretation that SAEs contain localised “OOD units”. Did the authors prove this? Does any previous paper discuss this interpretation? Why do the authors use their own OOD benchmark? The literature already discusses OpenOOD v1.5, which provides a clear explanation of why those datasets are used. How many models are used, and which ones serve as controls? What exactly is being tested here? What does “seed-idiosyncratic” mean?

What do the percentages in the abstract represent? The authors talk about confidence, but what confidence? They then discuss a collapse that is not directly related to the task as presented, and its definition only comes later. The presentation of the problem is not clear, and the abstract is hard to read and follow.

In the authors' own words, “The contribution is not a stronger detector but a tested boundary on what sparse-feature OOD analysis can support.” However, this statement does not highlight the benefits of this perspective. The abstract also does not mention the FPR95 or AUROC of the proposed method.

Introduction: The introduction begins without a proper definition of the task. What are the authors trying to do: OOD detection or just sparse autoencoder analysis? The authors also include statements between em dashes (“—”), such as “one the cited works do not themselves claim”. We can see this pattern of writing throughout a large part of the introduction, with statements that appear to contradict or weaken the claims made in the paper.

After reading the abstract and introduction, it is almost impossible to understand the goal of this paper. The paper has only one paragraph that discusses a kind of “interpretability” provided by SAEs, but the same text contradicts this in some parts. After these “explanations”, the paper quickly jumps to the contributions without providing any real introduction to the task, the state of the art, or the idea behind the method.

Regarding the contributions:

(1) A negative result on OOD units: The authors talk about confidence, which has not been properly explained at this point. Should I read the entire paper to understand this part? They also highlight a negative result that is really hard to understand in the way it is written.

(2) The collapse is reconstruction: The same problem appears here. This could even be considered the same contribution as the previous point.

(3) It is difficult to understand what each percentage or number means without any kind of explanation.

(4) The paper claims to provide a reusable audit, but the paper itself reads like a report without a natural flow between paragraphs, making the proposed procedure very hard to understand and reproduce. Also, the authors talk about a demo in the abstract and introduction, but there is no clear representation of it anywhere in the paper. What is the point of mentioning this if it is part of the contribution?

The paper claims that confidence collapse is a property of reconstruction and the classifier head. However, this is presented without sufficiently deep analysis, and the authors appear to be highlighting a phenomenon already known in the community regarding general autoencoders.

The paragraphs in many parts of the introduction are not connected, making the introduction really hard to understand and follow.

Related Work: Consider the following sentence: “We measure this instability in a vision-SAE OOD pipeline.” This defines a goal that was never properly introduced before, confusing the reader.

Setup and the Sparse Reconstruction Audit: This section is terrible in terms of presentation. It refers to different model weights and configurations without explaining why these particular models are used or the reasons behind their selection.

Then, a disconnected explanation of the encoders and classifier heads jumps, in the same format, to the SAEs and then to “Scorers”.

I want to emphasize this point because this section is particularly problematic. Suppose that the main contribution of the paper is the scorers, but these are never properly explained or highlighted. When the authors define the “Sparse Reconstruction Audit”, they provide a paragraph referring to a six-step protocol implemented in a Python file. The method is never explained in sufficient detail and is instead presented as a sequence of numbered steps without much more explanation.

This section shows several problems with the paper's presentation and polish, making it extremely difficult to understand the methodology from a scientific perspective.

Detection Performance : The datasets are never properly defined in the setup. Instead, they are introduced in the caption of Table 1, together with their citations. The table is hard to read and does not contain an average performance column. Its organisation is also bad because it is really hard to quickly identify which methods are proposed by the authors.

Also, the benchmark used does not follow the standard OOD evaluation protocols. The authors define their own collection of OOD datasets without sufficient explanation. There is no clear reason why they do not use OpenOOD, a benchmark specifically developed and studied for this task (Yang et al., OpenOOD: Benchmarking Generalized Out-of-Distribution Detection, NeurIPS 2022). The analyses are weak and do not contain anything much deeper than repeating the same numbers already presented in the tables. It is also hard to understand why the authors only report AUROC in the main table and do not include FPR95, which is a more demanding metric for OOD detection at a high true-positive rate.

The Confidence Collapse Is Reconstruction — and It Is Head-Dependent : After presenting the detection results, the authors introduce a new section without sufficient motivation. It feels like a completely new topic in the middle of the previous results.

Again, the method is explained too quickly, without explaining why the analysis is necessary or the reasoning behind it. An equation is simply introduced without much further explanation. After this, the paper presents an analysis of MSP but completely loses its focus on the SAE. For the reader, it becomes complicated to quickly understand the motivation behind this analysis.

Then, the paper mostly presents negative or weak results. These are highlighted by the authors, but they do not provide a sufficiently deep analysis of the SAEs and instead mix different results without a clear connection between them.

Conclusion :The conclusion does not properly conclude the work and remains open-ended. It does not provide the reader with a clear understanding of what the authors actually discovered or what the most relevant contribution of the paper is.

  • Zhang, J., et al. (2023). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. arXiv:2306.09301
Critical strengths

I could not identify any clear strengths in the current version of the paper. The presentation makes it extremely difficult to understand the scientific contribution, and the experimental results do not establish a sufficiently clear or novel insight. Although the paper reports several experiments, I do not consider their quantity alone to be a strength without a coherent scientific motivation and analysis.

Critical weaknesses

Please see my paper summary for more details:

  1. Poor presentation and writing: The paper is extremely hard to read and follow. The abstract, introduction, and methodology lack a clear scientific narrative, and many paragraphs are disconnected. The manuscript gives the impression of being insufficiently revised after substantial LLM-assisted writing, with numbers, claims, and findings presented without enough explanation.

  2. Unclear scientific contribution: It is difficult to understand what the main contribution of this paper is. The authors mix OOD detection, SAE interpretability, reconstruction analysis, and classifier-head sensitivity without properly connecting these objectives. The paper reads more like a collection of experimental results than a coherent scientific contribution.

  3. Weak methodology explanation: The proposed Sparse Reconstruction Audit is not sufficiently explained in the main paper. The authors mostly present a sequence of steps and references to Python files without providing enough motivation, mathematical explanation, or justification for the experimental choices.

  4. Weak experimental evaluation: The authors use their own collection of OOD datasets without sufficiently justifying why they do not follow established benchmarks such as OpenOOD v1.5. The main results are difficult to interpret, FPR95 is not included in the main detection table, and the proposed SAE methods do not consistently outperform existing baselines.

  5. Limited novelty and scientific insight: The main confidence-collapse effect is largely explained by reconstruction and classifier-head dependence rather than SAE-specific properties. PCA reproduces much of the observed behavior, raising questions about what the sparse representation actually contributes. The paper does not provide a sufficiently deep explanation of why these findings represent a novel contribution.

  6. Limited generalization: Several findings depend on the architecture, classifier head, or SAE configuration. Some attribution results do not transfer to the TopK SAE family, and the main confidence-collapse effect is not reproduced with the evaluated third-party SAE. This limits the generality of the conclusions.

Questions for authors
  • What prevents evaluating the proposed sparse-feature audit on standardized suites such as OpenOOD v1.5, reporting both AUROC and FPR95\text{FPR95} across standard near-OOD and far-OOD splits?
  • Can the authors provide a formal algorithmic box detailing the 6-step audit and the exact mathematical formulation of the scoring functions within the main text?
  • Given that confidence degradation under reconstruction is well-documented in classical autoencoders, what specific, novel dynamics are introduced uniquely by the sparsity constraints of SAEs?
4 comments
AuthorsOpenReview ↗

Part 1/4: OpenOOD v1.5 (Q1)

Thank you for the detailed review. We accept the criticism of the presentation and respond to it in part 4; parts 1 to 3 answer your three questions.

Q1. OpenOOD v1.5 with AUROC and FPR@95 on the near/far splits. We should have used it from the start. Our 13 sets started from the CIFAR-10 OOD sets of earlier post-hoc detection work (SVHN, LSUN, TinyImageNet, DTD, Places365), with sets added for semantic and style shift (ImageNet-O, -R and -Sketch); the submission cited OpenOOD but followed only its baseline definitions. ViT-B/16 is our main backbone because it is a supervised ViT with a native 1,000-class ImageNet head, which the head-width tests need, and an architecture OpenOOD v1.5 evaluates at ImageNet-1K. We ran it with OpenOOD's own image lists, near/far groups and metric code, and its test preprocessing on CIFAR-10 (each image resized to 32 px on its shorter side, bilinear, and center-cropped to 32×32 px). It uses OpenOOD's data, groups and metrics but not its training protocol, so the numbers should not be read against its leaderboard. The differences, all kept from the paper:

  • Frozen ImageNet-pretrained backbones (ViT-B/16, DeiT-B/16, ResNet-50) with a linear probe for the CIFAR-10 classes, instead of a network trained on CIFAR-10. Images then go through one 224-px transform for all backbones (bilinear resize, ImageNet normalization). The ViT-B/16 checkpoint's own transform (bicubic resize to 248 px, 224-px center crop, normalization 0.5) gives 85.3% instead of 81.0% ImageNet top-1 on 10,000 validation images; with features re-extracted and the SAE retrained under it, the submission's headline round-trip result is similar (75.5% versus 9.1% on ID, against 70.3% versus 10.7%).
  • At ImageNet-1K, mean-pooled features with the native head, and a Resize(224) crop instead of Resize(256).
  • One run per configuration (one backbone checkpoint and one probe; one SAE seed per CIFAR-10 configuration, three TopK seeds at ImageNet-1K); OpenOOD's CIFAR-10 numbers are means over three trained networks.
  • Our ViT-B/16 checkpoint was pretrained on ImageNet-21k before ImageNet-1K fine-tuning, unlike OpenOOD's ImageNet-1K-only ViT-B/16; since SSB-hard is built from ImageNet-21k classes, this may make ImageNet-1K near-OOD easier for every method.
  • Fits on our caches: 20,000 CIFAR-10 training images in five of the seven configurations (all 50,000 in the other two); 200,000 ImageNet training images, with KNN's bank on the first 50,000 of them.
  • Fixed hyperparameters (KNN k = 50, PCA rank 256, ViM's dimension rule), with no search on OpenOOD's validation sets.
  • MSP and Energy on the probe with temperature scaling fit on ID validation data (T = 1.9 to 3.3). With T = 1, as in OpenOOD, MSP's AUROC is 0.012 to 0.026 lower and Energy changes by at most 0.003; we report both.

Apart from the transform, the temperature and the fit size (refitting on all 50,000 images changes the ViT-B/16 baseline AUROCs by at most 0.022 for MSP and 0.009 for the others), we did not measure the effect of these choices. The full CIFAR-10 table is in our reply to eVUd; in brief (AUROC / FPR@95; "near" and "far" are means over the near- and far-OOD sets):

Near-OODFar-OOD
CIFAR-10, ViM (best)0.953 / 0.2010.976 / 0.097
CIFAR-10, MSP (T = 1 in brackets)0.908 / 0.302 (0.892 / 0.319)0.914 / 0.260 (0.897 / 0.281)
CIFAR-10, SAE reconstruction0.847 / 0.5500.895 / 0.356
ImageNet-1K, best baseline by AUROC0.741 / 0.713 (Mahalanobis)0.928 / 0.291 (ViM)
ImageNet-1K, SAE reconstruction (TopK, 3 seeds)0.706-0.710 / 0.740-0.7440.906-0.907 / 0.323-0.329

Neither SAE score beats the best baseline in any configuration (seven CIFAR-10 configurations on three backbones, and ImageNet-1K), and the revision says so instead of "competitive". The scorers are not our contribution: we evaluate standard scores computed on an SAE and do not propose a detector. In the full ViT-B/16 table (our reply to eVUd), the scores that use only distances to ID embeddings (KNN, Mahalanobis, PCA and SAE reconstruction) lie at 0.85-0.87 on near-OOD, while those that read the classifier's logits (MSP, Energy, ViM) reach 0.91-0.95 with the calibrated probe (MSP 0.89 at T = 1); the revision gives this as the likely reason the reconstruction scores trail. FPR@95 follows OpenOOD's code (the share of ID images flagged at the threshold that detects 95% of OOD images) and changes no conclusion: the lowest value is never an SAE score's. The OpenOOD near/far table, with an average column and the SAE-based scores marked, replaces Table 1, and both OpenOOD papers are cited. The run also revealed a resolution confound in our own benchmark (see the general comment), a further reason to switch.

References: Yang et al. (2022), OpenOOD, NeurIPS Datasets and Benchmarks; Zhang et al. (2024), OpenOOD v1.5, DMLR (arXiv:2306.09301).

AuthorsOpenReview ↗

Part 2/4: The audit as an algorithm (Q2)

Q2. Algorithm box with the exact scoring functions. Yes. The revision adds this box to Section 3 (quoted without the question that ends each step; the next paragraph paraphrases them); it states the submitted audit's rules unchanged:

Algorithm 1: Sparse Reconstruction Audit

Input: frozen encoder f; head h; SAE with encoder E and decoder D (columns d_i), trained on ID features; ID training and test sets; OOD test sets; a second head h₀ trained on the ID labels (a temperature-scaled linear probe).

For each input x: z = f(x), a = E(z), ẑ = D(a) and s(z) = max_c softmax(h(z))_c.

  1. Reconstruction score S(z) = ‖z − ẑ‖² / dim(z); AUROC of S, ID versus each OOD set.
  2. Confidence drop Δ(z) = s(z) − s(ẑ): AUROC of Δ, and the relative drop in mean confidence, 100·(E[s(z)] − E[s(ẑ)]) / E[s(z)], on ID and on each OOD set.
  3. Attribution: p_i = a_i²‖d_i‖² / Σ_j a_j²‖d_j‖²; the mean normalized entropy H(p) / log k over the k active units, and the modal-unit share, the fraction of a set's samples whose largest p_i falls on that set's most frequent top unit.
  4. Noise control: the relative confidence drop on each set when ẑ is replaced by z + ε, ε ~ N(0, σ²I), with σ² = E‖z − ẑ‖² / dim(z) over that set.
  5. Selectivity: the OOD and ID drops of step 2, and the OOD drop under h₀.
  6. Linear control: AUROC of the PCA reconstruction error and its relative confidence drop, for a rank-256 PCA fit on ID training features.

Decision rules: reconstruction-dominant if AUROC(S) ≥ 0.95, OOD drop ≥ 50% and |noise drop| ≤ 15%; diffuse if the mean normalized entropy exceeds the ID value by at least 0.10; localized if the modal-unit share is at least 3 times the ID value; OOD-selective if the OOD drop is at least 5 times the ID drop and at least 20 points; head-dependent if the OOD drop under h is at least 5 times that under h₀ and at least 20 points. Each verdict carries its margin, the signed distance from the threshold in the rule's unit: entropy for diffuse, multiples of the ID value for localized, points of drop from the larger threshold for OOD-selective and head-dependent, and for reconstruction-dominant the smallest of its three distances (AUROC or points).

Why each step is there: step 1 asks whether reconstruction error alone separates the sets; step 2 whether the confidence drop does; step 3 how evenly each sample's reconstruction is spread over its active units (diffuse) and whether OOD samples share the same top unit more often than ID samples (localized), two different questions; step 4 whether Gaussian noise of the same mean squared size lowers confidence as much; step 5 whether the drop is specific to OOD inputs and to the wide head; step 6 whether a linear basis with no sparsity does the same.

The revision reports every verdict we computed, including unfavorable ones. On ImageNet-O at 32 px, P = passes, F = fails, margins in brackets (20k: the paper's SAEs, fit on 20,000 CIFAR-10 images; all 50k: fit on all 50,000):

CIFAR-10 SAE (ReLU unless TopK)Recon.-dominantDiffuseLocalizedOOD-selectiveHead-dependent
Mean, 20k (paper)F (-0.01)F (-0.01)P (+1.33)P (+8.4)P (+16.0)
CLS, 20kF (-0.02)P (+0.016)F (-0.25)P (+10.5)P (+21.5)
Mean TopK, 20kF (-0.01)F (-0.087)F (-0.69)P (+32.8)P (+8.8)
CLS, all 50k, 3 seedsF (-0.01)P (+0.017 to +0.030)P (+1.05 to +1.81)P (+22.4 to +29.2)P (+7.8 to +14.4)
Mean, all 50k, 3 seedsF (-3.3 to -4.2)2 P, 1 F (-0.013 to +0.006)P (+0.15 to +1.57)P (+12.8 to +25.8)1 P, 2 F (-4.3 to +2.1)

In the last row the drop (46-47%) and AUROC (0.942-0.946) both fail and the margin is the drop's; elsewhere only AUROC fails (0.934-0.939). Applied to the OpenOOD table, head-dependence holds in 10 of 14 CIFAR-10 cells.

The detection scores, oriented so that higher means more OOD: MSP −max_c softmax(g(z))_c and Energy −log Σ_c exp g(z)_c, where g is the probe h₀ with temperature-scaled logits on CIFAR-10 and the native head h at ImageNet-1K; KNN, the negative cosine similarity to the 50th-nearest ID training embedding; Mahalanobis (Lee et al.), the smallest squared distance to a class mean under the shared within-class covariance; ViM (Wang et al.), α times the norm of z's residual outside the top D principal directions of ID features (D = 512, or 1,000 for ResNet-50; after ViM's origin shift), minus the logsumexp of the unscaled logits, with α the mean top logit over the mean residual norm on ID training data; SAE reconstruction S(z) as in step 1; SAE activation Mahalanobis ((a − μ)ᵀ (Σ + 10⁻⁶I)⁻¹ (a − μ))^½, with the ID mean μ and covariance Σ of the code; PCA reconstruction ‖(I − P_k P_kᵀ)(z − μ_z)‖ with the top-k ID principal directions P_k.

References: Lee et al. (2018), NeurIPS; Wang et al. (2022), CVPR.

AuthorsOpenReview ↗

Part 3/4: What sparsity adds (Q3)

Q3. What does the sparsity constraint add beyond classical autoencoders? We agree that a classifier's confidence dropping on reconstructed inputs is not specific to SAEs. We found no part of the confidence drop on a narrow ID that needs sparsity. PCA-256 fit on the same ID training set reproduces most of it: on ImageNet-O (ViT-B/16, mean pooling, ImageNet head), 73% versus the SAE's 70% at native resolution, 52% versus 62% at matched resolution, and 55% versus 54% in the CLS headline setting (against SAEs trained on all 50k images, PCA's drop is the larger one, 73% versus 53%). On ResNet-50 the reverse holds (SAE 14%, PCA 5% on ImageNet-O), and we have not tested whether sparsity causes it. What concentrates the drop on OOD inputs is fitting the reconstruction on a narrow ID. In the paper's comparison at a CLIP hook (an SAE spliced into the output of block 11 of CLIP ViT-B/32), SAEs trained with one recipe and data size on CIFAR-10 lower a 1,000-class zero-shot head's MSP by 60-64% on ImageNet-O versus 5-10% on ID, while the same recipe trained on ImageNet gives 31% versus 23-28%. The two arms end with different numbers of active units (181-197 against 313-323), so this comparison is not capacity-matched either. The paper's third arm, fit on Food-101 with the same recipe, ends with about as many active units as the CIFAR-10 arm (192-203 against 181-197) and lowers ID confidence more than OOD confidence (52-59% versus 41-44%), so in this comparison the effect needs an SAE fit on the audited ID. The size of the drop depends on the SAE: an ImageNet SAE with more capacity than ours (4,096 units, 262 active, about 1.15M training embeddings, against 2,048 units, about 132 active and 16,000 embeddings) lowers MSP on ImageNet validation images by only 1.8%, whereas PCA-256 fit on ImageNet still lowers it by 45% (61.4% when fit on CIFAR-10, close to the CIFAR-fit SAE's 60.6%). At similar active dimension (262 against 256), the ImageNet SAE's drop is thus far smaller than PCA's; we have not isolated whether sparsity, the overcomplete dictionary or reconstruction fidelity causes this. Its concentration on OOD inputs is a property of an ID-fit reconstruction read by a wider head, and its size depends on the SAE and the data it is fit on; neither is evidence about SAE features. Reconstruction error as an OOD score is classical (e.g., Denouden et al., 2018; Zhou, 2022); our two SAE scores are the sparse counterparts of reconstruction error and latent-space Mahalanobis distance, and the revised Related Work discusses this line. What we add is that the confidence drop used in sparse-feature analyses depends on the head's label space and on the data the reconstruction is fit on, with an audit that separates these from feature-level explanations.

The SAE matters in two places. (i) Unit-level questions (which units carry the signal, and whether they are stable across seeds) exist only for a sparse code. The results that units are seed-specific (confirming known SAE instability: Paulo and Belrose, 2026; Fel et al., 2025) and that five fixed units detect ImageNet-O within one SAE stay in the main text; the result that the largest-contribution units mostly carry ID evidence ("ID anchors") moves to an appendix. Resolution and SAE family matter here too: at native resolution ImageNet-O is diffuse for the ReLU SAE (entropy gap 0.128 over ID) but not for TopK (0.051, below 0.10), and at 32 px the ReLU SAE is localized rather than diffuse (part 2), so the revision gives both for every attribution result. (ii) At ImageNet-1K the TopK SAE is a better reconstruction-based detector than PCA (we ran no dense nonlinear autoencoder, so this does not separate sparsity from nonlinearity, and the ReLU SAE trails tuned PCA on far-OOD). We swept PCA's dimension from 16 to 760 (of 768) and took the best value on the test sets, which favors PCA; the best values, k = 640 (near) and k = 512 (far), lie inside that range. The TopK SAE still leads by 0.045 AUROC on near-OOD and 0.016 on far-OOD (smallest margins over three seeds). On CIFAR-10 the two are within 0.01 for mean pooling, and PCA is ahead for CLS pooling.

References: Denouden et al. (2018), arXiv:1812.02765; Zhou (2022), CVPR, pp. 7379-7387; Paulo and Belrose (2026), ICLR; Fel et al. (2025), ICML.

AuthorsOpenReview ↗

Part 4/4: Presentation, the abstract, and generalization

What the paper asks. When SAE features of a frozen vision model are used for OOD detection, what do the resulting signals measure? We test three readings: that SAE scores are strong detectors (on OpenOOD they are not), that individual units are stable OOD detectors (they are not stable across seeds), and that the drop in a classifier's confidence when its input embedding is replaced by the SAE reconstruction is evidence about SAE features. Section 5 turns to the classifier head because that drop depends on the head's label space and on the data the reconstruction is fit on, more than on the units. The benefit is an audit that tells a practitioner which of these explanations an SAE-based OOD signal supports before it is read as interpretability evidence.

Presentation. We agree with the problems you list, including the impression of text that was not sufficiently revised; the abstract, introduction and setup are rewritten. The main changes:

  • The abstract and introduction state the task first (post-hoc OOD detection with a frozen classifier), then the question, then the findings; the introduction ends with what a practitioner should check. The abstract reports OpenOOD AUROC and FPR@95 for SAE reconstruction and the best baseline.
  • The introduction summarizes the state of the art (post-hoc scores such as MSP, Energy, Mahalanobis, KNN and ViM, and recent SAE-based OOD work) before the contributions.
  • Em-dash asides are removed: each becomes a sentence or is cut. The introduction's aside "one the cited works do not themselves claim", which contradicted its first sentence, becomes: "The cited works claim neither the second nor the third reading."
  • "Confidence" is defined, before it is used, as the maximum softmax probability of the named head (for the round-trip drop, the frozen 1,000-class ImageNet head), and each percentage says what it is relative to.
  • References to code files, the demo and similar implementation details leave the main text.
  • Contributions 1 and 2 are merged (small unit sets detect OOD inputs within one SAE but differ across seeds and do not carry the drop), and the Related Work sentence you quote now follows from a stated goal.
  • The setup follows the box's order and gives the main datasets and models with their reasons before any result: ViT-B/16 as the main backbone (why: part 1); DeiT-B/16 (same architecture, other training recipe), ResNet-50 (a CNN) and DINOv2 ViT-B/14 (self-supervised, no ImageNet head) as other backbones; CLIP ViT-B/32, the model of the released third-party SAE, also for the controlled SAE comparison. The controls are matched noise, a PCA basis fit on the same data, a probe head matched to the ID, and a random projection and an untrained SAE, which lower confidence on every set. The main SAEs are ReLU and TopK, three seeds each. Results sections say what each number implies.
  • Sections 4 to 6 each open with why they follow from the previous one, Section 5 with the question it answers and an explained equation; each caption ends with a takeaway.
  • The conclusion states what we found and what it implies (three takeaways: our reply to eVUd, part 2).

Questions in your summary. That SAEs contain seed-stable OOD units is a hypothesis, shown neither by us nor by the cited works; the revised abstract states it as one we test. The works we cite report aggregate SAE signals of OOD inputs: that SAE features can be used for detection (Olson et al., 2025), class-specific activation profiles (Oh et al., 2026) and distinct sets of active features (Karmacharya et al., 2026); Chhetri et al. (2026) perturb representations along SAE concept vectors, which differs from our round-trip. These invite a unit-level reading, but none of them claims seed-stable OOD units. "Seed-idiosyncratic" meant specific to one training seed: a unit with no partner (correlation above 0.7) in SAEs trained with other seeds. The revision defines the term or drops it.

Generalization. The round-trip drop has the same direction on DeiT-B/16 and ResNet-50 (table in our reply to eVUd), but reaches 5 times the ID drop only in ResNet-50's far-OOD cell (8.3 times, but a 16.8-point drop, below the 20-point floor) and on both groups in no DeiT-B/16 or ResNet-50 configuration. With DINOv2 and a probe, the round-trip instead raises ImageNet-O confidence (by 16.3%). You are right that some attribution results do not transfer to TopK SAEs (part 3 names them), and the revision says for which SAE family each attribution result holds. The third-party SAE result is also a limit on generality: it was fit on ImageNet, the head's own domain, and in the paper's CLIP comparison (part 3), training on ImageNet largely removes the concentration on OOD inputs. Limitations states that the effect needs an SAE fit on the audited ID.

References: Olson et al. (2025), ICCV Findings; Oh et al. (2026), CVPR; Karmacharya et al. (2026), arXiv:2607.12094; Chhetri et al. (2026), arXiv:2606.16196.

Reviewer_jCukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sparse-autoencoder (SAE) based OOD detection on frozen ViTs. Its main SAEs (ReLU and TopK, three seeds each) are trained on mean-pooled ViT-B/16 embeddings of CIFAR-10. It reports four results.

First, individual SAE units are not stable across seeds: only ~8–10% have a >0.7-correlated partner in another seed. Fixed sets of up to five units add only ~1 point of confidence collapse beyond the plain encode→decode round-trip, and the largest-contribution units mainly gate ID confidence (“ID anchors”).

Second, under a frozen 1,000-class ImageNet head, the round-trip lowers max-softmax confidence by 70% on ImageNet-O versus 11% on ID (47% vs 11% with the native CLS input). Matched noise moves it by 7%, and a PCA round-trip reproduces the collapse (73%), so the effect belongs to ID-fit reconstruction rather than to the sparse code.

Third, the collapse depends on the head. At fixed ID and fixed SAE, restricting the head’s softmax to W random classes makes the OOD drop grow with W while the ID drop stays flat. Heads trained on the ID label space are inert. The effect is ~10% with ImageNet-100 as ID and 6.4% at ImageNet-1K with the native head. The authors name this “Mismatched Wide-Head Reconstruction Sensitivity.”

Fourth, on detection, SAE reconstruction error is competitive with but not better than published ViM and Mahalanobis baselines, is matched by PCA, and trails dense baselines at ImageNet-1K scale. The authors release a six-step audit. On a third-party Prisma CLIP SAE the effect is absent, and a controlled comparison of SAEs fit on CIFAR-10, ImageNet, or Food-101 at the same hook ties the effect to an SAE fit on the audited ID.

Critical strengths
  1. The authors condut complete controls and made a reproducible release. In addition to showing that a confidence collapse occurs, it systematically rules out the obvious alternatives. The paper does not stop at showing that a confidence collapse occurs; it systematically rules out the obvious alternatives. It separates the plain encode -> decode round-trip from unit ablation, and compares both against several noise nulls: isotropic noise matched to reconstruction error, logit-matched noise, residual-covariance noise, and residual swaps. It also compares against PCA, random-projection, and untrained-SAE round-trips, stratifies by base confidence, and replicates across seeds and two SAE families. The head-dependence claim is tested by a fixed-ID head-width sweep, heads trained at each width, an ImageNet-100 in-domain setting, and a second backbone, and the third-party SAE audit is followed by a three-arm comparison at the same hook. Beyond the results, the paper documents implementation pitfalls that can silently invalidate such studies, such as random masking that selects inactive units, the ViM sign convention, double normalization, and split-cache leakage. Every number traces to a released artifact, which makes the work easy to verify and build on.

  2. A useful practical caution about confidence-based diagnostics. The paper shows that a confidence collapse under reconstruction can be large and OOD-selective while reflecting neither the encoder nor any individual feature. Instead, it is a property of the system formed by the head and the SAE: the effect appears when a wide head whose label space does not match the ID reads the output of an SAE fit on that ID, and it disappears under a matched head or a general-purpose SAE. The controlled comparison of SAEs fit on CIFAR-10, ImageNet, and Food-101 at the same hook makes the role of the SAE’s training data especially clear. This gives practitioners a concrete warning: when a broad pretrained classifier is tested on a narrow deployment domain, a drop in confidence should not be read as evidence about the encoder or about specific features without first checking how the head and the SAE’s fit data contribute.

Critical weaknesses
  1. The central negative result targets a claim no prior work makes, and much of the rest is confirmatory. The headline result is that SAEs do not contain seed-stable, localized “OOD units.” But the paper acknowledges that this is a reading “the cited works do not themselves claim,” and that none of them claims a seed-stable vocabulary of OOD units. The findings are either consistent with prior work or outside the paper’s controls. The ID-anchor result is described as consistent with the class-profile ID signature of Oh et al.. Small fixed unit sets do detect OOD within a model (five units: 0.90–0.94 AUROC on held-out ImageNet-O), in line with Karmacharya et al.. The perturbation probe of Chhetri et al. is described as distinct from the round-trip analysis. The test setting also does not match the closest prior work. The main SAEs use mean-pooled tokens, while prior ViT SAE-OOD work uses CLS tokens. The CLS-trained SAE appears only as a control for the round-trip battery, and unit seed-stability is not reported for it. The result is therefore a preemptive boundary on a hypothetical reading, not a refutation of a published claim. With that removed, the remaining detection and stability results largely confirm Guan et al., Paulo & Belrose, and Fel et al. The title and abstract overstate the contribution relative to what is shown.

  2. The “mechanistic account” is a phenomenology, its named mechanism may be misattributed, and the headline magnitude comes from an artificial regime. The paper shows which configurations produce the collapse and rules out its own two candidate explanations, softmax arithmetic and head/ID-subspace misalignment, but offers no explanation in their place. A simpler account fits the data: the ID-fit reconstruction removes the directions that support the head’s confident prediction on an OOD input, and width matters only because a random W-class subset is more likely to contain that predicted class, which is plausible given that ImageNet-O was built to elicit confident wrong predictions from ImageNet heads. This coverage account is consistent with three observations: at W = 10 the drops have near-zero means with very large spread and matched noise raises confidence; with ImageNet-100 as ID the drop saturates from W = 300 to 1,000 (10.3% to 10.4%); and with heads trained at each width the excess over noise levels off beyond W = 100. If coverage rather than width is the lever, both the name of the phenomenon and the paper’s mechanistic framing need revision. In addition, the headline 70% comes from a 1,000-class head applied to mean-pooled features it was never trained on (80.8% top-1 agreement with the true forward pass), with upscaled 32×32 images and an SAE trained on 16k embeddings, while the more realistic settings show a modest or absent effect: 47% with the native CLS input, about 10% with ImageNet-100 and trained heads, 6.4% versus 3.2% at ImageNet-1K with the native head (failing the paper’s own 5× criterion), and no effect for the third-party SAE.

Questions for authors
  1. Using the existing head-width sweep, can you split ImageNet-O samples at each W by whether the full head’s top-1 class is inside the W-class subset, and report the round-trip drop for each group? Please do the same for the heads trained at each width.

  2. Which concrete published inference fails under your controls? Please name a specific claim from prior work that your findings contradict, rather than one they are consistent with or do not test. If none exists, would you reframe the paper as a preemptive boundary rather than a refutation?

  3. Can you report unit seed-stability for SAEs trained on CLS tokens? This is the setting used by the closest prior work. Please report the fraction of units firing on > 1% of samples that have a >0.7-correlated cross-seed partner, and the stability of small OOD-selected unit sets.

  4. The main SAE is trained on 16k embeddings, and the L0-bisection scheduler targeted 32 active units but stopped at k = 132 because of the λ\lambda ceiling. Why use a subset rather than all 50k CIFAR-10 training images? Is k = 132 a chosen design point or a by-product of the ceiling? Does unit stability change when the SAE is trained on the full set?

  5. At narrow W, matched Gaussian noise increases OOD confidence by 15–53%. What explains this? Is the restricted-softmax baseline at small W reliable enough to support the low-W end of the “monotonic” trend?

3 comments
AuthorsOpenReview ↗

Part 1/3: The coverage account (Q1 and W2)

Thank you for a careful and constructive review. We tested your coverage account directly; it explains part of the effect.

Q1. Split by whether the full head's top-1 class is in the W-class subset. ImageNet-O, ViT-B/16 mean pooling, the paper's CIFAR-10 SAE, native resolution (the submitted configuration; the headline setting is at the end of this part). Fixed head: the paper's 10 random subsets per width, with coverage defined by the native head's top-1 class. Heads trained at each width: 5, 5, 5, 3 and 1 heads at W = 10, 30, 100, 300 and 1000, with coverage defined by the top-1 class of the trained 1,000-class head. Relative drop in mean MSP under the round-trip, for samples whose top-1 class is in / not in the subset:

WFixed head: share inFixed head: in / outTrained heads: share inTrained heads: in / out
101.0%20.0% / -3.2%0.8%26.6% / 2.8%
303.3%38.6% / 11.9%2.5%36.3% / 11.1%
10010.4%53.9% / 27.4%9.8%37.0% / 26.7%
30029.6%65.6% / 44.5%33.3%41.6% / 35.1%
1000100%70.3%100%40.3%

Top-1 classes are taken from the clean embedding, before the round-trip. Across the 10 subsets the covered drop has a standard deviation of 2.2 to 7.4 points (uncovered 5.2 to 23.1), and matched noise moves the covered group by 1% to 8% and the uncovered group by -16% to -2%.

Coverage matters: with the fixed head, the covered samples carry 47% of the lost confidence at W = 100 (10% of samples) and 68% at W = 300 (30% of samples). With the trained heads the covered drop is nearly flat from W = 30 (36-42%), as your account predicts, and the ImageNet-100 saturation you note (10.3% at W = 300, 10.4% at W = 1000) also fits it; we did not run the split there. So does the trained-head excess over noise you cite (17, 27, 34, 37 and 34 points), which levels off beyond W = 100; the revised abstract says so, and limits "grows monotonically" to CIFAR-10 as ID. But it is only part of the story: with the fixed head the covered drop still rises with W (20% to 70%), and the uncovered drop rises in both cases.

At fixed width the rank, under the full head, of the best class in the subset matters strongly: at W = 100 the drop is 54% when that class is the full top-1, 49% for ranks 2-5, -3% for ranks 6-20, and -94% (confidence rises) for ranks 21-100. Wider subsets contain higher-ranked classes, which explains part of the width trend; the rise at fixed rank does not follow from it. Because ImageNet-O was selected to fool ImageNet heads, we checked the 12 other OOD sets: on all 12, for all six SAEs, at W = 100 the top-1 rank loses more confidence than ranks 21-100, and the top-class logit loss tracks the drop across sets (Spearman ρ 0.95 to 0.98). Our untested hypothesis is that wider subsets also contain the top class's strongest competitors, whose evidence the round-trip removes too; consistent with it, masking the full head's top-5 classes reverses the drop (MSP rises by 65%), and the covered samples' base MSP falls from 0.98 at W = 10 to 0.49 at W = 1000. In the headline setting (rules fixed in advance) covered samples also lose more (44-45% against 34-37% at W = 300) and their drop still rises from 19-20% at W = 30 to 53-54%, but masking the top-5 classes only removes the drop there; the drop rises at every width step (three seeds).

Why the top classes lose evidence. Two tests on all 13 OOD sets (ImageNet-O shown):

  • Let r be the reconstruction residual and u the unit direction of the top class's centered head weight. The component (r·u)u carries 2.3% of the residual's energy but reproduces 60 of the 70 points of the drop, and the rest of the residual raises confidence (-5.6%). For this SAE the same split holds on all 13 sets: the component carries at least three quarters of the drop with at most 5% of the residual's energy, and the rest raises confidence by 3% to 19%.
  • On ImageNet-O, 38.5% of the top-class logit comes from the part of the embedding outside the top-256 principal directions of CIFAR-10 embeddings, against 3% on CIFAR-10. On ImageNet validation images (all 50,000) that part alone predicts the label at 64.7% top-1, and the CIFAR-fit SAE round-trip lowers the head's top-1 accuracy from 80.0% to 58.8% and its MSP by 60.6%. The second half of this test, more off-subspace evidence under the ImageNet head than under the CIFAR-10 probe, fails on 5 of the 13 sets (SVHN, MNIST, CIFAR-100 and both LSUN variants), so this part of the explanation is only partly supported.

So your account captures a real part of the effect, and the revision presents the whole as an empirical account rather than a mechanism, with these numbers. It drops the name of the effect, and we propose a new title; part 2/3 gives the audit verdicts behind that change. As in the submission, every new number in our replies traces to a script and a result file, and the revised anonymous supplementary will include them.

AuthorsOpenReview ↗

Part 2/3: Framing (Q2, W1) and the realistic settings (W2)

Q2 / W1. Which published inference fails under our controls? None. The works we cite do not claim seed-stable OOD units, and our results are consistent with them. Your reading of the rest is fair too: the stability and detection results largely confirm Guan et al. (2023), Paulo and Belrose (2026) and Fel et al. (2025), the ID-anchor result agrees with Oh et al. (2026), and five-unit detection with Karmacharya et al. (2026); Chhetri et al. (2026) perturb along SAE concept vectors, unlike the round-trip we test. What we consider new is the audit of what the round-trip confidence drop measures. The revision frames the paper as a tested boundary (a preemptive boundary, in your words) on how sparse-feature OOD signals can be read, not as a refutation: the introduction says which readings the cited works do not make, and the conclusion presents the audit as a diagnostic. We propose a title and abstract to match.

W2. The headline comes from an artificial regime. We agree. The revised abstract and Section 5 lead with a setting that addresses all three factors you list: native CLS input, SAEs trained on all 50,000 CIFAR-10 images, and ImageNet-O brought to the 32 px of CIFAR-10, which removes the resolution mismatch (both sets are still upscaled 32-px images). There the round-trip lowers MSP by 54% (53-54% over three seeds) on ImageNet-O versus 5% on ID (4.9-6.3%), with matched noise at 1%. At native resolution the same SAEs give 26% against the same ID drops (4.1 to 5.3 times), so the OOD-selective rule passes in two seeds by about one point and fails in one (-5.7), and the paper's CLS SAE (47% versus 11%, 4.2 times) fails it (-9.0). On OpenOOD (all images at 32 px, one SAE seed) it is 43% on near-OOD and 26% on far-OOD versus 5%.

The audit's verdicts in this setting are why we drop "Reconstruction, Not Localization" from the title: reconstruction-dominant fails narrowly (AUROC 0.936-0.938 against 0.95), and localized passes (19-21% of ImageNet-O samples share the same largest-attribution unit, against 4-5% of ID samples, 4.1 to 4.8 times the ID value). Checked since (rule fixed in advance), the unit has a partner above 0.7 in all six seed pairs, and in four it is the other seed's own top unit; but it is dense, and zeroing it adds only 1.6-2.5 points to the drop. Diffuse also passes; it measures something else, how evenly each sample's attribution is spread over its active units, so samples can be diffuse while many share the same top unit. OOD-selective passes (+22 to +29 points), and head-dependence passes in all three seeds (the CIFAR-10 probe loses 8-9%; margins +7.8 to +14.4 points). With mean pooling and the same all-50k SAEs at 32 px, head-dependence fails in two of three seeds (-4.3 in both).

Each factor separately:

  • Upscaled 32-px images. With ImageNet-O at 32 px, the drop is 62% versus 11% on ID with mean pooling, with matched noise at 2%; with CLS input it is 66%, larger than the 47% at native resolution.
  • SAE trained on 16k embeddings. With SAEs trained on all 50,000 images (40,000 training embeddings after the split instead of 16,000; three seeds), the drop is smaller but more concentrated on OOD inputs: 53% (51-54%) versus 5% on ImageNet-O with mean pooling and 26% versus 5% with CLS input at native resolution, and 46% and 54% at 32 px. These ReLU SAEs are also less sparse (about 180-200 active units, against 122-133), so data and sparsity change together. The TopK pair holds k = 131 fixed: 66% versus 1% (mean pooling, native resolution), against 76% versus 4% with 16k embeddings. The size therefore depends on the SAE; the revision reports both.
  • Mean-pooled input to a CLS-trained head. With CLS input the OpenOOD drops are 11.5% / 55.1% / 39.3% (ID / near / far).

The realistic settings stay modest. With ImageNet-100 as ID the effect is about 10%, and the third-party SAE, fit on ImageNet, shows none (our reply to 8Wfp, part 4, explains why). At ImageNet-1K, where, as in the paper, mean-pooled features go to the native head, it is small: with the TopK SAE (seed 42), 1.9% on ID versus 6.2% (near) and 7.1% (far) on OpenOOD; with the paper's ImageNet-1K ReLU SAE, 3.2% on ID versus 7.2% and 6.3% on OpenOOD and 6.4% on ImageNet-O (2.0 times; margin -13.6 points), which fails our own 5× criterion; the revised abstract states these sizes. The criterion also fails on most new OpenOOD cells: with its 20-point floor it passes in 2 of 14 CIFAR-10 cells and in none of the four ImageNet-1K cells (two SAEs); the ratio alone reaches 5 in 4 of 14 cells, and on both groups of a configuration in 1 of 7 (general comment, item 3); the paper's configuration gives 4.7 and 4.0. The revision keeps the criterion, reports its verdict for every cell, and gives the ratios.

References: Guan et al. (2023), ICCV; Paulo and Belrose (2026), ICLR; Fel et al. (2025), ICML; Oh et al. (2026), CVPR; Karmacharya et al. (2026), arXiv:2607.12094; Chhetri et al. (2026), arXiv:2606.16196.

AuthorsOpenReview ↗

Part 3/3: Unit stability (Q3), training-set size and k (Q4), the noise null (Q5)

Q3. Unit stability for CLS-trained SAEs. The submission used mean pooling for detection and the audit (one SAE for both); on OpenOOD it gives SAE reconstruction a higher AUROC than CLS in all six cells of our three mean/CLS pairs (MSP and Energy mostly favour CLS). The ImageNet head was trained on CLS, and the closest prior work uses CLS, so the revision leads with it. The paper's measures, three seeds per setting, ranges over seed pairs. A unit is frequent if it fires on at least 1% of samples, and a partner is a unit in another seed with correlation above 0.7. "20k" and "50k" are the CIFAR-10 images used (16,000 and 40,000 training embeddings after the split).

SettingFrequent units with a partnerFive OOD-selected units with a partnerAUROC of each seed's five-unit set
Mean, ReLU, 20k (paper)7-8%20%0.90-0.94
Mean, TopK, 20k16-18%0-40%0.67-0.83
Mean, ReLU, 50k9-11%0-40%0.90-0.95
Mean, TopK, 50k23-25%0-20%0.78-0.81
CLS, ReLU, 20k6-8%0%0.96-0.98
CLS, TopK, 20k14-15%0%0.74-0.81
CLS, ReLU, 50k9-11%0%0.97-0.98
CLS, TopK, 50k21-23%0-20%0.80-0.83

(AUROC on the held-out half of ImageNet-O at native resolution, where the resolution cue of the general comment, item 2, may contribute.) The partner shares count frequent units only, which is why the mean-pooling 20k rows differ from the all-unit values of the submission's main text (8.2-8.8% ReLU, 9.7-10.4% TopK); the frequent-unit values were in its appendix. Within each SAE family, CLS units are at least as seed-specific as mean-pooling units, and each seed finds a different five-unit set; these sets detect ImageNet-O with AUROC 0.67 to 0.98 across settings, yet fixed sets of up to five units add only about one point of confidence drop beyond the round-trip: they detect, but do not carry the effect.

Q4. Training-set size and k. The 20,000-image subset was a runtime cap, not a methodological choice: we set it when we first built the pipeline, to keep a complete run within about four hours on a single 16 GB GPU, and kept it. We retrained on all 50,000 CIFAR-10 training images. The instability claim holds in all eight settings above (mean frequent-unit partner share below 30% and five-unit share below 50%, thresholds written with this analysis), although partner shares rise somewhat with more data (by up to 7 points for TopK); the size of the round-trip effect does change (part 2/3). k = 132 was not a chosen design point: the L0-bisection scheduler targeted 32 active units and stopped at the λ ceiling, so the code is about four times denser than that target (we have no run at an L0 of 32), and the TopK SAEs fix k = 131 to match it (k = 119 for CLS, from the CLS ReLU SAE). With 50k images the same scheduler stops at an L0 of about 180-200.

Q5. Why matched noise raises confidence at narrow W, and whether the low-W end is reliable. For most ImageNet-O samples, no class in a random 10-class subset clearly dominates (for ViT mean, mean restricted MSP 0.2 to 0.4 in the four lowest of five quintiles). Isotropic noise in the embedding shifts each class logit by a random amount with standard deviation σ‖w_c‖, which spreads the tied logits apart and raises the top softmax. These shifts are correlated across classes; replacing them with independent shifts of the same size reproduces the effect almost exactly. At W = 10, independent shifts versus embedding noise give -17.7% vs -17.6% for ViT mean, -35.9% vs -35.8% for ViT CLS, -56.1% vs -56.2% for DeiT CLS, and -27.4% vs -27.5% for DeiT mean. The rise is concentrated on samples with flat subset logits, shrinks or reverses on confident ones, and grows with the noise scale. It is larger on ImageNet-O because the noise is matched to each set's reconstruction error: its standard deviation σ, the root-mean-square reconstruction error per coordinate, is 1.7 to 3.7 times larger there than on ID.

With 1,000 random subsets per width, the low-W end is reliable for mean pooling. For ViT mean the drop is -3.2%, -2.0%, 4.1%, 10.5%, 20.6%, 35.9%, 50.2%, 56.8%, 63.7% and 70.3% at W = 5, 10, 20, 30, 50, 100, 200, 300, 500 and 1000. Every adjacent step is a resolved increase (its 95% interval over class subsets excludes zero) for all six mean-pooling ViT SAEs (ReLU and TopK, three seeds each) and for DeiT with mean pooling; for the paper's SAE and DeiT mean the same holds for the excess over matched noise. The submission's W = 10 standard deviation (23.1) came from one outlying subset; over 1,000 subsets it is 10.5. For CLS pooling with the 20k SAE at native resolution the drop dips at small W (to W = 10 for ViT, to W = 20 for DeiT) and is resolved as rising only from W = 20 (excess over noise from W = 30); in the headline setting it rises at every step. Section 5 states the trend accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results and the revision

We thank the reviewers for careful reviews. Two asked for a standard benchmark, and all three found parts of the paper hard to follow. Below are the new results, including where they contradict the submission, what stays the same, and the revision; the reply to each reviewer gives the details.

New experiments

  • OpenOOD v1.5 (Zhang et al., 2024): its image lists, near/far-OOD groups and metric code (AUROC, FPR@95), with our frozen ImageNet backbones and linear probes instead of its training protocol, so the numbers are not comparable with its leaderboard (our reply to 8Wfp lists every difference). CIFAR-10 is the in-distribution (ID) data and, as in OpenOOD, every image is first resized to 32 px on its shorter side and center-cropped to 32×32 px. Seven feature configurations on three backbones (ViT-B/16, DeiT-B/16, ResNet-50), each with a near and a far group, give 14 cells; ImageNet-1K is run with ViT-B/16.
  • The round-trip confidence drop (relative drop in a frozen classifier's mean maximum softmax probability, MSP, when each embedding is replaced by its SAE reconstruction) on the same sets.
  • ImageNet-O resized to 32 px like CIFAR-10; SAEs trained on all 50,000 CIFAR-10 training images (40,000 embeddings after the split; three seeds); for jCuk, the coverage split, unit stability and the noise null; for 8Wfp, a PCA dimension sweep.

What changed in our conclusions

  1. SAE-based detection is not competitive: SAE reconstruction is below the best baseline in all 14 CIFAR-10 cells and trails it at ImageNet-1K by about 0.03 AUROC on near-OOD, so "competitive" is removed (reply to eVUd, part 1).
  2. Our CIFAR-10 benchmark had a resolution confound: several OOD sets were used at full resolution, and on OpenOOD's Places365 list SAE reconstruction falls from 0.995 to 0.834 AUROC once its images are at 32 px, so the OpenOOD table replaces Table 1 (reply to eVUd, part 2).
  3. Our own 5× check mostly fails: the round-trip drop is larger on OOD than on ID inputs in every cell, but at least 5 times larger on both groups of a configuration, as the check was fixed before the run, in only 1 of 7 CIFAR-10 configurations and in no ImageNet-1K cell (replies to eVUd and jCuk).
  4. In the setting we now lead with, the audit contradicts the submitted title: with CLS input, ImageNet-O at 32 px and SAEs trained on all CIFAR-10 training images, the drop is 54% versus 5% on ID, yet the audit's reconstruction-dominant rule fails narrowly and its localized rule passes (its dense top unit has a partner in every seed pair), so we drop "Reconstruction, Not Localization" from the title (reply to jCuk, part 2).
  5. The size of the drop depends on the SAE: at native resolution with mean pooling, SAEs trained on all CIFAR-10 training images give 53% on ImageNet-O against 70% for the paper's SAE, an SAE fit on ImageNet lowers MSP on ImageNet validation images by only 1.8%, and we have not run a capacity-matched comparison (replies to jCuk, part 2, and 8Wfp, part 3).
  6. A head matched to the ID is less affected, not inert: on ImageNet-O a CIFAR-10 probe loses 4.5% with the paper's SAE at native resolution, and with SAEs trained on all 50,000 images 7.1% (native resolution, one SAE) and 7.8% to 10.0% (32 px, six SAEs) (32-px verdicts: reply to 8Wfp, part 2).

What stays the same

The question, the six-step audit with its thresholds, and every submitted experiment (in the main text or an appendix). Each change answers a request in the reviews, and most claims that change become weaker. Two do not: at ImageNet-1K the TopK SAE now beats PCA with a tuned dimension as a detector (reply to 8Wfp, part 3), and in the new headline setting the drop is smaller but more concentrated on OOD inputs (about 10 times the ID drop, against 6.6 in the submission). Changes are marked in blue in the revised PDF.

What the revised paper contributes: an audit with fixed rules and controls that tells a reader whether an SAE-based OOD signal reflects reconstruction error, the classifier head or specific units, and the finding, on OpenOOD and three backbones, that the round-trip drop depends on the head's label space and on the data the SAE is fit on more than on its units.

The revision

  • Proposed title, subject to the area chair's agreement: "A Controlled Audit of Sparse-Autoencoder OOD Signals in Vision Transformers". The effect's long name becomes the plain "round-trip confidence drop".
  • The abstract and Section 5 lead with the setting of item 4, with the submitted numbers beside it; Section 5 gives all five audit verdicts.
  • Abstract, introduction and setup rewritten; an algorithm box; the OpenOOD table; the unit-level section cut to one stability table, the result that five fixed units detect ImageNet-O within one SAE, and short summaries of the other unit analyses.

Reference: Zhang et al. (2024), OpenOOD v1.5, DMLR (arXiv:2306.09301).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.