acceptodds
Under review as a conference paper at ICLR 2027

When Weak Features Hide: Separating Statistical Detection from Sparse Autoencoder Recovery

Abstract

To understand weak features missed by sparse autoencoders (SAEs), we distinguish three questions: whether a statistical test detects a feature, whether a decoder can represent it, and whether training recovers it. We analyze covariance tests that refit coordinate-wise variances and derive a closed-form visibility factor that vanishes for features aligned with the detector's coordinate axes. Such signals can be absorbed into the fitted variances, creating a test-specific blind spot that does not restrict an unconstrained SAE decoder's representational capacity. We also derive a held-out alignment certificate under rank-one Gaussian covariance and recovery guarantees for decoder families that explicitly preserve a data-estimated anchor. In a prespecified synthetic control, an isotropic-null-calibrated covariance statistic has power 0.081 for an axis-aligned feature and 1.000 for a dense feature, while matched TopK SAEs recover both in all eight seeds. A post-hoc correlation-based sensitivity analysis yields the same qualitative contrast. A separate post-hoc experiment holds the architecture and non-anchor initialization fixed: training-PCA anchors yield 183/192 recoveries versus 21/192 for random anchors. The same comparison yields no recovery-rate advantage on the tested injected GPT-2/Pythia residual backgrounds. These results separate detector-specific invisibility from decoder feasibility and clarify the assumptions required for certified recovery.

Primary area:
learning theory
Keywords:
sparse autoencoders, covariance testing, statistical learning theory, nuisance parameters, sample complexity, sparse coding, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D64COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that covariance tests which refit coordinate-wise variances can miss axis-aligned features, even when sparse autoencoders can represent and recover them. It also derives alignment certificates and guarantees for preserving PCA-estimated directions, with positive synthetic results but no recovery advantage on the tested language-model residual backgrounds. The paper theoretically characterizes a detector specific blind spot and guarantees alignment under explicit constraints (known as anchor constraints), while empirically showing that SAEs can recover detector-invisible features and that PCA anchoring helps in synthetic settings but offers no recovery-rate advantage on the tested language-model residuals.

Critical strengths

I think it is nice that this paper separates statistical detection, decoder representability, and actual recovery through training. The problem is well motivated

Critical weaknesses

Limited novelty PCA eigenvalue/eigenvector transitions are classical spiked-covariance results, and finite-sample alignment bounds follow from standard eigenvector perturbation theory. Here, Theorem 4 applies that machinery and then freezes the estimated direction; Theorem 5 adds a bounded angular displacement. These are useful constructions, but do not establish a new recovery threshold or explain SAE learning dynamics.

Insufficient substantive comparison with dictionary learning Spielman–Wang–Wright establish recovery of sparsely used dictionaries, and Arora–Ge–Moitra give guarantees for learning incoherent, overcomplete dictionaries. Those works address identifying unknown dictionary directions from sparse observations—a more direct version of the motivating feature-recovery problem. Their assumptions and algorithms differ from modern SAEs, so they do not automatically subsume this paper, but the authors should explain precisely what unresolved problem their results address.

Detector is Narrow Contribution The visibility coefficient is a clean calculation, but exact axis blindness follows because the alternative belongs to the composite diagonal-covariance null. The potentially distinctive contribution is its explicit geometric characterization and juxtaposition with SAE recovery—not the general observation that nuisance fitting removes statistical information.

Limited practical evidence Synthetic successes occur in favorable settings, while the matched language-model residual experiments show no recovery-rate advantage. The theory’s relevance to realistic SAE failures therefore remains unestablished.

Experimental Limitations

The experimental evidence is too narrow to establish a substantive advance in SAE feature recovery. The main synthetic experiments demonstrate that a deliberately restricted covariance detector can fail where an SAE succeeds, but do not explain realistic SAE failures; the anchor experiments largely test preservation of a direction already recoverable by PCA, without establishing a clear benefit beyond PCA itself. Moreover, the matched GPT-2/Pythia experiments show no recovery-rate advantage, and the evaluation lacks a systematic study of weak targets amid stronger competing features, correlated activations, and varying noise and sparsity. Although the authors transparently report negative results and limitations, the experiments primarily illustrate the conceptual distinction rather than establish a broadly useful recovery mechanism.

Questions for authors
  1. What is the precise theoretical novelty beyond established dictionary-learning and spiked-covariance results? Which theorem provides a new recovery insight, rather than an application of standard perturbation bounds or a consequence of preserving an estimated direction?

  2. What practical SAE procedure corresponds to the proposed detector? Why should blindness of a test that refits coordinate variances help us understand failures of reconstruction-trained SAEs?

  3. What does SAE training add beyond PCA in the anchor experiments? Can you demonstrate improved feature recovery, latent-code accuracy, or reconstruction relative to PCA alone and an unconstrained PCA-initialized SAE?

  4. Does the approach recover weak features that are not leading principal components? How does performance change with stronger competing features, correlated latent activations, and nonorthogonal dictionary directions?

  5. Why does the anchor advantage disappear on language-model residuals? Is the failure attributable to initial PCA estimation, subsequent optimization, or the usage-based atom-selection rule?

  6. How does the subspace-alignment certificate translate into individual feature recovery? What prevents a certified subspace from containing the target without any atom—or its activations—meaningfully representing it?

  7. Can the main detector–decoder comparison be repeated with the correctly calibrated correlation statistic and equal sample budgets? This would separate the conceptual distinction from calibration and data-budget differences.

  8. How robust are the empirical conclusions across dimension, dictionary width, sparsity, and training budget? Can you report activation-level recovery alongside atom cosine, with chance baselines adjusted for dictionary size?

4 comments
AuthorsOpenReview ↗

Response to Reviewer D64C (1/3): theoretical novelty and the role of the detector (Q1–Q2)

Thank you for the detailed review and for recognizing the value of separating statistical detection, decoder representability, and recovery through training. Your eight questions map closely onto analyses we have run; we answer them in order and include one new experiment run for this response (Q7).

Q1 / Novelty relative to spiked-covariance and dictionary-learning theory. The paper’s theoretical contribution is a set of results that hold uniformly where existing tools are usually applied to one estimator at a time:

Detector side (Proposition 1, Theorem 1, Corollaries 2–3). We express the evidence available to a detector class as the norm of the feature score after projecting out that class’s nuisance directions. For the frame-diagonal class this yields a closed-form geometric factor, 1−∥B⊤w∥441-\lVert B^\top w\rVert_4^4, which ties evidence loss to the detector frame rather than to the feature. This makes a precise, testable prediction—rotating BB at fixed strength changes independence power but not label power—which our rotation experiment confirms (r=0.977r=0.977 with the predicted coefficient; label power ≡1\equiv 1). Recovery side (Theorems 4–5). The new insight is not a perturbation bound by itself but its uniformity: for anchor-preserving decoder families, the same recovery guarantee holds for every objective, capacity, optimizer, and initialization that respects the constraints. This is exactly what lets the guarantee hold without modeling SAE learning dynamics. Empirically, the exact-anchor family recovers the target in 48/48 rows across three objectives, two capacities, and two geometries (minimum cosine 0.9989; family-wise certified lower bound 0.9405), and the trainable bounded-drift family also passes 48/48 (certified lower bound 0.8543).

Spielman–Wang–Wright and Arora–Ge–Moitra establish identifiability of unknown dictionaries for specific algorithms under sparsity and incoherence assumptions. Our question is complementary: what a specified detector’s failure implies about recoverability by reconstruction-trained SAEs, and which structure makes recovery objective-agnostic. We have added this comparison to the related-work section.

Q2 / What practice the detector corresponds to, and why its blindness matters. The frame-diagonal detector formalizes a common auditing step: standardize or whiten each coordinate, then look for remaining dependence, for example to decide whether activations or SAE residuals still contain an unexplained feature. Its blindness matters because a silent check of this kind is easily read as “no feature here” or “the feature is unrecoverable.” As you note in your review, the distinctive contribution here is the explicit geometric characterization of this blind spot and its juxtaposition with SAE recovery; this is precisely what the paper sets out to provide. Our matched control shows that this inference fails exactly for features aligned with the check’s coordinates: the detector’s power is at the nominal level, yet a reconstruction-trained TopK SAE recovers the same feature in every seed (Q7 below).

AuthorsOpenReview ↗

Response to Reviewer D64C (2/3): beyond PCA, and weak non-leading features (Q3–Q4)

Q3 / What SAE training adds in the anchor experiments. The anchor experiments test whether training preserves a data-estimated direction while fitting a full sparse dictionary—the prerequisite for any anchor-based recovery guarantee. Two comparisons address your question directly:

Against random anchors, with everything else matched (data, seed, optimizer, non-anchor initialization; 192 pairs): training-PCA anchors yield 183/192 recoveries versus 21/192 (paired difference +0.844, 95% CI [0.797, 0.891]). Against an unconstrained PCA-initialized SAE (dose-response control, eight independent anchor groups): every mechanism recovers 8/8, but the unconstrained SAE lets residual atoms leak into the anchor direction (maximum residual–anchor dot product 0.734 versus ≤ 0.102 for every constrained mechanism) and has the largest and most variable held-out loss (0.1414 ± 0.0077 versus ≤ 0.1373). The structured family therefore delivers controlled leakage and stable reconstruction on top of recovery.

Unlike PCA, the SAE also learns sparse codes for a full overcomplete dictionary. We tested this directly on SynthSAEBench-16k (768 dimensions, 16,384 features, correlated and hierarchical firing; five seeds): BatchTopK reaches MCC 0.754, versus 0.629 for FastICA and 0.171 for PCA, and learns an atom aligned at cosine ≥ 0.80 for 73 of 1,280 pre-specified target features, versus 1 for FastICA and 2 for PCA. Our pipeline reproduces the public SynthSAEBench checkpoints (MCC 0.753 vs. 0.757–0.758).

Q4 / Weak non-leading features, competing and correlated features. Our residual-background experiments address this directly. We inject a known feature into real GPT-2-small and Pythia-70M residual-stream activations, where it must be learned among many naturally interfering, correlated, and non-orthogonal features. One injected direction is the 32nd principal direction of the empirical background, the weakest of the 32 retained components rather than a leading one. At λeff=0.55\lambda_{\mathrm{eff}}=0.55, the trained dictionaries contain an atom aligned with this non-leading direction (cosine ≥0.80\ge 0.80) in 8/8 (PCA anchor) and 6/8 (random anchor) GPT-2 conditions, and in 8/8 and 6/8 Pythia conditions. These alignments are not chance coincidences: random 64-atom dictionaries in this space reach cosine 0.80 in 0 of 20,000 draws. Recovering such a feature therefore does not require it to be a leading principal component.

AuthorsOpenReview ↗

Response to Reviewer D64C (3/3): LM residuals, certificate, equal-budget experiment, and robustness (Q5–Q8)

Q5 / Why the anchor advantage disappears on LM residuals. We re-scored the archived runs (48 matched pairs per model; no retraining), counting conditions with absolute target cosine ≥ 0.80 at three points: the initial anchor, the best final atom, and the usage-selected atom.

GPT-2, training-PCA anchor: 40/48 initial anchor, 32/48 best final atom, 0/48 usage-selected atom. GPT-2, random anchor: 0/48, 30/48, 0/48. Pythia, training-PCA anchor: 44/48, 41/48, 1/48. Pythia, random anchor: 0/48, 21/48, 1/48.

The answer to your question is clear: initial PCA estimation is accurate on both backgrounds (40/48 and 44/48), and the trained dictionaries usually still contain a well-aligned atom (32/48 and 41/48 with PCA anchors). What fails is the pre-specified usage-based rule for choosing which atom to evaluate, which almost never selects that atom. The aligned atoms are not chance coincidences: random 64-atom dictionaries in this 32-dimensional space reach cosine 0.80 in 0 of 20,000 draws. The bottleneck on LM residuals is therefore primarily atom selection rather than anchor estimation.

Q6 / From the subspace certificate to individual features. Theorem 3 is deliberately objective-agnostic and subspace-level: it certifies any frozen decoder, however trained, from a held-out reconstruction comparison. Atom-level guarantees are the role of Theorems 4–5, which add the anchor-preserving structure that pins the target to a specific atom. The two results are complementary: Theorem 3 audits an arbitrary trained subspace, while Theorems 4–5 guarantee a specific atom when the structure is imposed.

Q7 / Calibrated statistic and equal sample budgets. We ran exactly this comparison. Following a protocol fixed before execution, the correlation statistic and TopK now use the same 20,000 examples per condition, and the detector is calibrated at that sample size (empirical size 4.5–5.1% across four diagonal-Gaussian variance scales). At the primary strength (λeff=0.55\lambda_{\mathrm{eff}}=0.55, 8 seeds):

Detector power on independent draws: 4.4% (142/3200) for the axis-aligned feature vs. 100% (3200/3200) for the dense feature. Detector rejections on the exact matrices used to train TopK: 0/8 (axis-aligned) vs. 8/8 (dense). TopK recovery (decoder cosine ≥ 0.80): 8/8 (axis-aligned) and 8/8 (dense).

On the very same training matrices, the detector never flags the axis-aligned feature, yet TopK recovers it in every seed. The axis-aligned power stays at the nominal level at all three strengths (4.4%, 4.4%, 4.8%), as the theory predicts. The contrast is therefore not a product of calibration or of unequal sample budgets.

Q8 / Robustness, activation-level recovery, chance baselines.

Dictionary width and sparsity. In the independent held-out grid (d,k)∈{(8,1),(16,2),(32,4)}(d,k)\in\{(8,1),(16,2),(32,4)\} with eight new seeds, axis-aligned features are never recovered less often than matched dense features: 8/8 vs. 8/8, 8/8 vs. 8/8, and 6/8 vs. 5/8. The widest configuration is budget-limited rather than geometry-limited: continuing the same trajectories to 16k steps gives 8/8 (axis) and 7/8 (dense). Objectives and capacities. The anchor-preserving families pass 48/48 rows across three objectives (MSE, MSE+ℓ1\ell_1, MSE+ℓ2\ell_2) and two capacities each, and all nine dose-response mechanisms recover 8/8. Strength. Across the strength sweep, TopK recovery rises with strength in both geometries, while the independence detector keeps its axis–dense gap. Chance baselines adjusted for dictionary size. For E4-v2 (16 atoms in 8 dimensions), random dictionaries reach mean maximum cosine 0.68; TopK reaches 0.970 (axis) and 0.978 (dense). For the LM experiments, random dictionaries never reach the 0.80 threshold (0/20,000).

Thank you again; these questions helped us sharpen both the theory’s role and the empirical evidence. We would be happy to answer any further questions at any time, and we look forward to your response.

Reviewer_D64COpenReview ↗

Response

Thank you for the clarifications and additional experiments, which address the calibration concern and strengthen the evidence for atom-level recovery. However, the central concerns remain: the anchor guarantees largely preserve an already estimated direction by construction, and the experiments do not clearly establish recovery of genuinely weak, non-leading features or benefits attributable to the proposed constraints. These leave the novelty and practical significance insufficiently established, so I will maintain my score.

Reviewer_YqDEOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors of the manuscript are interested in weak features missed by sparse autoencoders (SAEs). The paper addresses more carefully three questions that are often mingled: can a statistical test detect a weak feature? can a sparse encoder represent it? and can training recover it? The main experimental contribution is a matched synthetic control (m = 8, a single sparse feature), which highlights that "invisible to this detector" does not necessarily imply "unrecoverable by an SAE".

Critical strengths
  • the paper proposed a clean, conceptual decomposition based on the three questions

  • the authors are honest about various limitations encountered during the study(e.g. the failed gate at d=32, k=4; the 19/48 certification rate)

  • the relevance of the matched-pair design with paired differences and seed-clustered resampling

Critical weaknesses
  • Theorem 1 and the first three corollaries are classical; since Cardoso's works on ICA, the decompostion dependence = correlation + non-Gaussianty is explicited. See the term C(Y) at the bottom of page 1181 in "Dependence, Correlation and Gaussianity in Independent Component Analysis" (JMLR, 2003)

  • I don't see how Theorem 3 could be used as a practical certificate, as it requires the oracle comparison PwP_w and knowledge of σ2\sigma^2 and τ\tau; it is actually a certificate for a reconstruction gap with respect to some oracle

  • the model considers n >> m superposition, but the experiments on recovery use a single rank-one spike on isotropic noise in m=8. By itself, PCA recovers it 96/96. This does not cover the actual phenomena that make SAE recovery a hard problem : many features interfering, correlated features, hierarchical features ("A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders", NeurIPS 2025). why not use a benchmark with a realistic scale in order to run into the real issues, such as  SynthSAEBench-16k?

  • PCA, ICA and vanilla SAE baselines are necessary

Questions for authors

See the weakness list

3 comments
AuthorsOpenReview ↗

Response to Reviewer YqDE (1/2): relation to Cardoso (2003) and the role of Theorem 3

Thank you for the careful review and for summarizing the paper’s central claim precisely: that “invisible to this detector” does not imply “unrecoverable by an SAE.” We also appreciate your recognition of the three-question decomposition, the matched-pair design, and our transparent reporting. Below we address each point and add new evidence run for this response.

  1. Relation to Cardoso (2003). Thank you for the pointer; we have added the citation and a discussion to the related-work section. Cardoso’s C(Y)C(Y) characterizes the correlation part of statistical dependence for a full Gaussian model. Our question is different: which part of that evidence survives when a specified detector refits nuisance parameters in a fixed frame, and how that evidence relates to recovery by a reconstruction-trained SAE. The detector-class projection (Proposition 1) answers this for any regular nuisance family. For the frame-diagonal class it yields a coefficient, 1−∥B⊤w∥441-\lVert B^\top w\rVert_4^4, that depends on the detector frame rather than on the feature itself. This frame dependence is what our experiments manipulate. Rotating BB at fixed strength changes independence power in line with the predicted coefficient (r=0.977r=0.977), while label power stays at 1. Corollaries 2–3 then place this detector on the same evidence scale as label and fixed-background tests, which is what lets us state the detection pattern our control tests.

  2. Theorem 3 as a practical certificate. Theorem 3 is designed as an audit that applies to any frozen decoder, however it was trained: given a hypothesized direction, a held-out reconstruction comparison converts into an alignment bound. This is the practical use case. Practitioners routinely ask whether a trained SAE captures a direction of interest (for example, a probe direction), and the theorem answers that question without assumptions about the training objective. Atom-level recovery guarantees come from Theorems 4–5. For anchor-preserving families, these hold uniformly over objectives, capacities, optimizers, and initializations. Empirically, both the exact-anchor and bounded-drift families recover the target in 48/48 rows across three objectives, two capacities, and two geometries.

AuthorsOpenReview ↗

Response to Reviewer YqDE (2/2): new equal-budget experiment, SynthSAEBench-16k, and baselines

  1. The single-spike control and PCA. The single-spike control is deliberately minimal. It holds every feature property fixed and changes only the feature’s geometry relative to the detector frame, so any difference between detection and recovery can be attributed to that geometry. The 96/96 PCA figure you cite comes from a separate appendix comparison of warm-start initializations, not from the main detector–decoder control. We also ran a new experiment addressing whether the contrast depends on calibration or on sample size. Following a protocol fixed before execution, the calibrated correlation detector and TopK now use the same 20,000 examples per condition. At the primary strength (8 seeds): Detector power on independent draws: 4.4% (142/3200) for the axis-aligned feature vs. 100% (3200/3200) for the dense feature. Detector rejections on the exact matrices used to train TopK: 0/8 (axis-aligned) vs. 8/8 (dense). TopK recovery (decoder cosine ≥ 0.80): 8/8 (axis-aligned) and 8/8 (dense). On the very same training data, the detector never flags the axis-aligned feature, yet TopK recovers it in every seed. The detector’s empirical size is 4.5–5.1% across four diagonal-Gaussian variance scales, so this is a properly calibrated test. On SynthSAEBench-16k (768 dimensions, 16,384 features with correlated and hierarchical firing), we compared methods on the same data across five seeds. Our training and scoring pipeline first reproduces the public checkpoints (BatchTopK MCC 0.753 vs. 0.757–0.758 for the five released seeds). In this overcomplete regime, BatchTopK reaches MCC 0.754, compared with 0.629 for FastICA (768 components, converged in all seeds) and 0.171 for PCA (768 directions). Among 1,280 pre-specified target features, BatchTopK learns an atom aligned at cosine ≥ 0.80 for 73, versus 1 for FastICA and 2 for PCA. Once features outnumber dimensions, PCA and ICA cannot substitute for SAE training.

  2. PCA, ICA and vanilla baselines. The appendix reports a vanilla SAE baseline: vanilla initialization recovers 45/96 conditions versus 89/96 with a spectral warm start. In the structurally matched comparison (Section 5.3), training-PCA anchors yield 183/192 recoveries versus 21/192 for random anchors under identical data, seeds, and non-anchor initialization (paired difference +0.844, 95% CI [0.797, 0.891]). Against an unconstrained PCA-initialized SAE, the anchor-preserving family recovers equally well (8/8) while keeping residual-atom leakage into the anchor at most 0.102, versus 0.734 for the unconstrained SAE, and reconstruction more stable. On SynthSAEBench-16k, PCA and FastICA baselines are reported in point 3 above. We hope these additions, especially the equal-budget result on identical training data, address your concerns, and we would appreciate it if you would reconsider your assessment in light of them. We would be happy to answer any further questions at any time, and we look forward to your response.

Thank you for the responses and for your effort. The new calibrated experiment is a very good piece of news, but the axis-aligned power is 4.4%, which is the test’s size. For me, this confirms that the axis-aligned alternative lies inside the null family. Also, 73/1,280 targets is a 5.7% recovery rate, which is nothing to be excited about. I am happy to increase my initial rating, but there are still a lot of things to strengthen in order to validate the ideas in this manuscript.

Reviewer_u6UGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theoretical and empirical mechanisms underlying feature detection, representation, and reconstruction failures in sparse autoencoders (SAEs), focusing specifically on "weak features" that are missed during dictionary learning. The authors propose an analytical framework pairing a prespecified synthetic control with a post-hoc correlation-based sensitivity analysis.

The paper claims four main contributions: 1) A local projection calculus formulated for a specified detector frame, 2) Exact decoder-feasibility theoretical guarantees, 3) A held-out alignment certificate for frozen subspace representations, and 4) Recovery guarantees for exact and bounded-drift anchor-preserving families.

The analysis is executed across both synthetic controls and empirical model activations (GPT-2 and Pythia residual streams). While the theoretical foundation is thorough and rigorous, the overall exposition is imbalanced: notation is dense without sufficient upfront explanation, and several formal theorem blocks mix mathematical statements with informal commentary.

Critical strengths

-Compelling and Principled Theoretical Motivation: Characterizing why and when sparse autoencoders fail to recover weak or low-magnitude features is a fundamentally important open question in mechanistic interpretability. The paper provides a rigorous mathematical lens on feature suppression and dictionary recovery guarantees.

-Carefully Designed Validation Framework: The combination of exact recovery bounds, bounded-drift guarantees, and synthetic-to-empirical controls (e.g., residual background analyses) provides a solid, multi-faceted theoretical and experimental evaluation of the problem.

Critical weaknesses

-Thin Preliminaries and Heavy, Undefined Notation: The paper is difficult to parse because complex, dense notations are introduced before the necessary background is established. Key symbols (such as N∗N^*, λ\lambda, and BB in lines 45–47) and concepts appear well ahead of their formal definitions, creating an unnecessarily high barrier to entry for readers.

-Unarticulated High-Level Impact and Practical Value: The authors do not adequately emphasize or clarify the broader practical importance of the proposed theory. It remains unclear how practitioners should translate these alignment certificates and recovery guarantees into concrete improvements in SAE training recipes, dictionary sizes, or downstream interpretability pipelines.

-Drafting Imbalances and Conflation of Proofs with Commentary: The presentation lacks structural polish. Formal blocks (such as Theorem 2, Proposition 2, Theorem 4, and Theorem 6) conflate formal mathematical results with explanatory remarks in their final sentences. In addition, sections such as Subsection 5.1 consist of a single standalone sentence, indicating an unevenly developed manuscript.

Questions for authors

-- Mathematical Rigor and Formalism in Theorems: In Theorem 2, Proposition 2, Theorem 4, and Theorem 6, the final sentences read like explanatory commentary rather than formal mathematical assertions. Which specific clauses represent the formal result, and can the narrative remarks be moved to discussion paragraphs? In lines 189–192, please provide a clearer, step-by-step breakdown of the toy example: why is C_B=0 in the first case, and why does C_B=1/2 in the second? In line 275, what is the precise mathematical definition of the phrase "remaining dictionary D_0 is free"?

-- Notation and Contextual Clarifications: In the abstract, what specifically is meant by the phrase "GPT-2/Pythia residual backgrounds"? In lines 45–47, please define N^*, λ, and B upon their initial appearance. In line 191, what does E_"ind" formally represent? In line 205, what exact partition does "This split belongs to" refer to? In line 292, what is the formal definition of the "E4-v2 control," and what are lines 295–296 intended to convey?

--Could the authors explicitly articulate how these theoretical bounds inform practical dictionary learning? What actionable guidelines emerge for avoiding weak feature collapse when training SAEs on real models?

--other comments: In line 160 and line 164, GPT-2 activations and the term "GPT-2 bridge" appear abruptly without prior framing. Please provide proper narrative context for why and how these empirical activations are introduced. Lines 285–288 appear disjointed from the preceding text; please ensure a smooth conceptual transition. Subsection 5.1 currently contains only a single standalone sentence. Please expand it to provide a thorough discussion of the results in Figure 2, or merge it with adjacent subsections to ensure balanced formatting.

2 comments
AuthorsOpenReview ↗

Response to Reviewer u6UG (1/2): notation, abstract, and toy example

Thank you for the careful reading and the line-level questions. We agree that the notation and structure raised the entry barrier. Below we answer each question and state the specific revision made in the updated PDF.

Notation (L45–47, L191, abstract).

N⋆N^\star is the number of samples a specified test needs to reach prescribed type-I and type-II error levels. λi\lambda_i is the strength of feature ii: the variance λi2\lambda_i^2 that the feature adds along its unit direction (for a centered Bernoulli feature with amplitude aa and firing probability pp, we use λeff=ap\lambda_{\mathrm{eff}}=a\sqrt{p}). BB is the orthonormal frame in which the detector may refit coordinate variances. EindE_{\mathrm{ind}} (L191) is the leading-order evidence available to that detector, i.e. NN times its minimum local KL divergence: Eind(i)≍N(1−∥B⊤w∥44)λi4E_{\mathrm{ind}}(i)\asymp N(1-\lVert B^\top w\rVert_4^4)\lambda_i^4 (Corollary 2). “GPT-2/Pythia residual backgrounds” are residual-stream activations of GPT-2-small (layer 6) and Pythia-70M (layer 3) on WikiText-2, PCA-whitened on training sequences; we inject a known feature direction into them, so the background is empirical while the target is known.

We now define N⋆N^\star, λ\lambda, and BB where they first appear in the introduction, define EDE_D at the start of the detector-hierarchy subsection, and rephrase the abstract accordingly.

Toy example (L189–192). Here CBC_B denotes the geometric factor 1−∥wB∥441-\lVert w_B\rVert_4^4; it differs from Ci(B)C_i(B) in Theorem 1 by the factor 1/(4σeff4)1/(4\sigma_{\mathrm{eff}}^4), and we renamed it κB(w)\kappa_B(w) to remove the clash.

wB=(1,0)w_B=(1,0): ∥wB∥44=14+04=1\lVert w_B\rVert_4^4=1^4+0^4=1, so κB=0\kappa_B=0. The spike only inflates the variance of coordinate 1, which the detector refits. wB=(1/2,1/2)w_B=(1/\sqrt2,1/\sqrt2): each coordinate contributes (1/2)4=1/4(1/\sqrt2)^4=1/4, so ∥wB∥44=1/2\lVert w_B\rVert_4^4=1/2 and κB=1/2\kappa_B=1/2. The spike adds off-diagonal covariance λ2/2\lambda^2/2, which a diagonal refit cannot absorb.

We also moved the example to follow Corollaries 2–3, after EDE_D (and hence EindE_{\mathrm{ind}}) is defined, and spelled out both computations in the text.

AuthorsOpenReview ↗

Response to Reviewer u6UG (2/2): theorem statements, specific lines, and practical value

Formal statements vs. commentary (Theorem 2, Proposition 2, Theorem 4, Theorem 6). The formal content is:

Theorem 2: for every unit ww there exist DD and a 1-sparse zz with Dz=αwDz=\alpha w. Proposition 2: for every ww and every unit vv, some feasible decoder has active atom vv. Theorem 4: the sine bound and the resulting cosine bound, with the stated probability. Theorem 6: the two rate statements (i) and (ii).

The remaining sentences (scope reminders such as “This is a representation statement…”, “This is not an end-to-end trained-SAE guarantee”) are commentary. We moved them into Remarks after each block.

L205. “This split” refers to the detection pattern just described: a present feature visible to the label and fixed-background covariance tests but invisible to the frame-diagonal test. It is a property of the nuisance family Mind,B\mathcal M_{\mathrm{ind},B}, not of sparse decoders. The revised sentence states this explicitly.

L275. D−0D_{-0} denotes all dictionary columns except column 0. “Free” means unconstrained and optimized jointly with column 0 to minimize population reconstruction error, with code coordinate 0 clamped to z−pz-p. Proposition 3 now writes the joint minimization explicitly.

L292–296.

E4-v2 is the second, pre-specified version of our matched detector–decoder control. Axis and dense loadings share the same dimension (m=8m=8), prevalence, strengths and seeds; we measure detector power (label, covariance, independence) and TopK recovery on both. Section 5 now opens with this definition. L295–296: the label test uses the known activation, so its power cannot depend on BB. Label power staying at 1 under every rotation therefore checks that the rotation changed only the coordinates used by the independence statistic, not the signal.

L160/L164 and L285–288. The introduction now explains why empirical residual backgrounds are used: known directions are injected into GPT-2/Pythia residual-stream activations so that the analysis is also exercised on realistic, non-isotropic backgrounds. We renamed the “GPT-2 bridge” to “residual-background experiments” throughout. The disjointed passage at L285–288 has been removed from the main text.

Section 5.1. We merged it with the primary-result subsection, which now introduces Figure 2 and describes each of its panels in the text.

Practical value. We added an “Implications for practice” paragraph to the Discussion with three takeaways:

Statistical invisibility should not be read as SAE failure without checking which nuisance parameters the detector refits and in which frame. A check that refits coordinate variances is blind to features aligned with its coordinates, yet in our matched control TopK recovered such a feature in all eight seeds (mean cosine 0.970). Theorem 3 gives practitioners a held-out audit for a hypothesized direction in a frozen SAE, converting a reconstruction comparison into an alignment bound. Anchoring an atom to a training-PCA direction raised synthetic recovery from 21/192 to 183/192 under matched initialization, and Theorems 4–5 identify structural constraints under which such an anchor provably survives training.

Thank you again; these comments made the paper easier to follow. We would be happy to answer any further questions at any time, and we look forward to your response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.