acceptodds
Under review as a conference paper at ICLR 2027

Beyond Retraining Stability: Auditing Duplication Dependence in Vision Sparse Autoencoders

Abstract

Sparse autoencoders learn directions from a chosen activation corpus, making their features sensitive to both model representations and the distribution of training examples. Can a direction recur across retrainings yet depend on repeated exposure to particular images? We investigate this question through controlled near-exact duplication in activation corpora from frozen vision transformers. Our audit compares each reference direction’s displacement after corpus intervention with its variability across unchanged-data retrainings. We contrast full removal with copies-only removal, which retains the original image, and include size-matched placebos for both. Directions that are stable under unchanged-data retraining can lose their closest counterparts when repetitions are removed. Above the estimated detection threshold, copies-only removal reproduces most of the full-removal effect and identifies substantially overlapping directions. Wider dictionaries exhibit lower detection thresholds across independent paired experiments, although absolute thresholds vary. These thresholds remain audit-dependent, and calibration diagnostics reveal tail inflation among stable latents, precluding an unconditional false-discovery-rate guarantee. Furthermore, combinations of retrained features recover much of the reference activation signal, distinguishing dedicated-direction loss from information loss. Our findings show that retraining stability alone does not establish robustness to duplication in the SAE training corpus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse autoencoders, mechanistic interpretability, data duplication, feature stability, dictionary learning, vision transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oa8MOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1(−1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether sparse autoencoder directions that recur across training seeds depend on repeated images in the SAE training corpus. The proposed audit compares decoder-direction changes after removing duplicate groups or only their extra copies against per-latent retraining variability and size-matched random removals. Experiments on frozen vision-model activations report greater sensitivity at low duplication doses in wider dictionaries and weaker effects for naturally occurring duplicates.

Critical strengths
  1. The paper addresses an important question about the reliability of SAE features.
  2. The experimental study is extensive and includes useful controls.
  3. The authors clearly discuss the limitations of their findings.
Critical weaknesses
  1. The paper lacks an overview figure illustrating the design, including the removal conditions and retraining comparisons.
  2. The paper assigns candidate latents to groups based on duplicate-specificity, then measures how their decoder directions change after removal and retraining. However, all 42 groups are modified together, so a latent associated with one target could shift because repetitions of other images were removed. The experiment therefore does not isolate whether the assigned target’s repetitions are responsible for that latent’s change.
  3. The proposed method accounts for each feature’s usual variation across retrainings when assessing the effect of duplicate removal. However, the baseline comparison also changes which features are tested, with the proposed method using a smaller, more selective set. IMO, a comparison using the same candidates is needed since testing fewer features can itself make detections easier.
  4. The main experiments track changes in decoder directions after duplicate removal, but this does not directly show how features respond to new images. The functional test has insufficient coverage to compare dictionary widths, while the recovery analysis is limited to near-duplicate variants of the same original image. A broader held-out evaluation is needed to determine whether the geometric width trend is accompanied by changes in feature behavior.
Questions for authors

Refer to weakness section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oa8M

Thank you for the constructive review. Each weakness asked for something we could test, so we ran a new, fully independent experiment: new targets, new crops, a rebuilt activation cache, and a new reference dictionary with its own 40 same-data retrainings (20,762 images, widths 4,096 and 16,384). The paper's procedure replicates on it: 110 to 113 flags at width 4,096 and 215 to 230 at 16,384, zero on every placebo, zero on 10 held-out null retrainings, and a paired width ratio of 11.3 [6.7, 18.3].

1. Overview figure. Agreed. The revised PDF adds a design figure: frozen backbone, cached activations, reference SAE, 40 same-data retrainings, the four removal arms with their placebos, and the comparison of each latent's displacement with its own retraining variability.

2. All 42 groups are removed together. We now remove one target group at a time: 8 targets (two at each k in {8, 16, 32, 64}), two seeds each, each with a size-matched random-removal control (32 new SAEs at width 4,096). In each run we measure the removal-minus-control displacement of the latents assigned to the removed target, and of the latents assigned to the other 41 targets.

  • Latents of the removed target: +0.252 when only its group is removed, versus +0.254 when all 42 groups are removed together (share 0.99).
  • Latents of the other 41 targets: -0.0001.

Per target the single and joint gaps agree closely (for example +0.489 and +0.474 vs +0.476 for one k = 32 target). One k = 8 target, below this experiment's threshold, moves in neither condition. A latent's displacement is therefore explained by removing its own target's repetitions, not other groups'. One caveat: with about 10 affected latents among 403 tests, a single-group run does not reach the BH threshold, so this evidence is the continuous displacement rather than flags.

3. Comparison on the same candidates. On the same 403 candidates, we replaced the per-latent standardised null with a pooled raw-distance null (no per-latent conditioning). It flags 0 latents in every removal run at both widths, versus 110 to 113 (4,096) and 215 to 230 (16,384) with the paper's null, while both flag zero on placebos. With the candidate set held fixed, detection comes from conditioning each latent on its own retraining variability, not from testing fewer latents.

4. How features respond to new images. We built a held-out probe the SAEs never saw: 16 fresh near-duplicate views of each target, plus 2,000 unrelated Food-101 images outside the corpus. Each target's top reference latent is matched to the retrained latent whose activations correlate best on half the probe; we then evaluate that fixed match on the other half and report 1 minus the held-out correlation. Coverage is now 41/42 targets at width 4,096 and 37/42 at 16,384 (previously 11/42 and 0/42). No null and no FDR are involved.

Removal-minus-placebo functional displacementk = 2k = 4k = 8
width 4,096+0.008+0.107+0.214
width 16,384+0.299+0.415+0.364

The functional change tracks the geometric one: it rises with dose, and at low doses it is much larger in the wider dictionary. The geometric width trend is therefore accompanied by a change in how the features respond to held-out images.

All four analyses will be added to the revised appendix. We hope these results address your concerns and would be grateful if you would consider them in your assessment.

Reviewer_ZfcSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether features learned by vision Sparse Autoencoders (SAEs) can remain stable across retraining while depending on duplicated images in the training corpus. The authors propose an instability-controlled auditing framework that measures feature displacement after removing duplicated samples, using repeated training on unchanged data as a baseline for natural optimization variability.

The main contributions and validation include:

  • Duplication-dependence auditing: The authors compare full removal and copies-only removal against size-matched placebo interventions, demonstrating that some apparently stable SAE features are sensitive to repeated exposure to specific images.
  • Dictionary-width analysis: Experiments across different SAE dictionary sizes show that wider dictionaries require fewer duplicated samples to exhibit detectable duplication dependence. This trend is reproduced across five independent paired experiments.
  • Feature recovery analysis: The authors show that combinations of features from retrained SAEs can recover much of the original activation signal, suggesting that the disappearance of individual feature directions does not necessarily imply information loss.

Overall, the paper provides empirical evidence that retraining stability alone is insufficient to establish the robustness or corpus independence of SAE features. It also highlights important limitations in the statistical calibration of the proposed audit, particularly regarding false discovery rate guarantees.

Critical strengths
  1. The paper presents a well-controlled framework for distinguishing duplication-dependent SAE features from ordinary retraining instability. The authors use 40 same-data retrainings, four removal and placebo conditions, and independent paired experiments. These controls provide convincing evidence that repeated exposure can influence the formation and persistence of dedicated feature directions.

  1. The findings have important implications for SAE-based interpretability. Wider dictionaries require fewer repeated examples to exhibit detectable duplication dependence. This challenges the assumption that more fine-grained SAE features necessarily represent robust concepts. The results also motivate further research into more reliable feature discovery and validation methods.
Critical weaknesses
  1. The statistical reliability of feature-level discoveries remains uncertain. The proposed audit relies on a pooled empirical null, candidate filtering, and Benjamini–Hochberg testing. However, the authors acknowledge tail inflation and the absence of formal FDR guarantees. Although the placebo controls provide supporting evidence, the sensitivity of the reported discovery counts and detection thresholds remains unclear. This limits confidence in the quantitative conclusions.

  1. The generalizability and interpretation of the dictionary-width effect remain limited. Naturally occurring near-duplicates produce much weaker effects than artificially introduced duplicates, with no flagged features. Moreover, dictionary width may affect feature matching, baseline stability, and detection sensitivity. It is therefore difficult to fully distinguish changes in learned feature structure from changes in audit detectability. These limitations raise questions about the practical applicability of the findings.
Questions for authors

The following questions concern issues that materially affect my assessment. Clear responses, supported by additional analyses where feasible, could address these concerns and lead me to raise my score.


Statistical calibration and reliability of feature discoveries

The paper acknowledges tail inflation in the pooled empirical null and the absence of formal FDR guarantees. The placebo experiments are reassuring, but the robustness of the reported discoveries remains uncertain.

Could the authors evaluate the sensitivity of the main results to alternative null calibration or more conservative multiple-testing procedures? Do the conclusions remain consistent when accounting for dependencies introduced by candidate selection and shared intervention runs?

This would help establish whether the reported discovery counts and detection thresholds are robust to the choice of statistical procedure.


Separating dictionary-width effects from detection sensitivity

The replicated dictionary-width effect is an important finding. However, wider dictionaries may also change feature-matching behavior, baseline instability, and detection sensitivity. The continuous displacement and targeted win-rate analyses partially address this concern.

Could the authors further distinguish changes in learned feature structure from changes in detection sensitivity? For example, does the width-dependent trend persist under alternative feature-matching criteria or evaluation procedures that better control for baseline instability?

This distinction is important for determining whether the observed width effect primarily reflects SAE learning behavior or the sensitivity of the proposed audit.


Generalizability to naturally occurring duplicates

The experiments with naturally occurring near-duplicates show much weaker effects than the controlled experiments. No features are flagged in these settings. This raises questions about how broadly the findings apply to real-world SAE training corpora.

Could the authors clarify whether the weaker results are mainly due to lower duplication counts, differences in activation-space similarity, or other dataset characteristics? A comparison controlling for duplication count and similarity would help isolate these factors.

This would help assess the practical relevance of the findings to realistic SAE training corpora.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZfcS

Thank you for the careful and constructive review. To answer your questions with new evidence, we ran a sixth fully independent experiment (new targets, new crops, rebuilt cache, new reference dictionary with its own 40 same-data retrainings and 10 held-out retrainings; 20,762 images; widths 4,096 and 16,384). All analyses below use this experiment.

Q1. Robustness to the statistical procedure. We re-ran the audit under five procedures on the same candidate sets (403 latents at 4,096, 416 at 16,384).

ProcedureFlags, 4,096Flags, 16,384Placebo flagsHeld-out nulls flaggedk*(4,096)/k*(16,384)
Paper (pooled null + BH)110-113215-23000/10, 0/1011.3 [6.7, 18.3]
Stability-stratified null + BH118-128212-2240; one run 20/10, 1/108.0 [4.7, 14.6]
Benjamini-Yekutieli0172-17600/10, 0/10wide only
Holm (FWER)0000/10, 0/10n/a
Raw pooled null + BH0000/10, 0/10n/a
  • The stratified null, which pools null scores only within a latent's stability tercile, removes the tail inflation we reported: among the most stable latents P(p <= .01) falls from 0.0248 to 0.0062 (width 4,096) and from 0.0258 to 0.0083 (16,384). Discoveries and the width ratio survive this correction.
  • Benjamini-Yekutieli is valid under arbitrary dependence, including that induced by candidate selection and shared runs. Under it the narrow dictionary flags nothing while the wide one flags 172-176, the same direction as the width effect.
  • Under Holm nothing is flagged at either width. The per-latent evidence consists of many moderately strong signals rather than a few extreme ones. We will state this plainly, keep discovery counts as procedure-dependent summaries, and rest the quantitative claims on the placebo-adjusted continuous displacement below, which uses no null calibration or multiple-testing correction.
  • On dependence across shared runs: the independent unit for the width claim is the experiment, and the paired ratio exceeds 1 in all six independent experiments (five in the paper, plus this one).

Q2. Learned structure vs. audit sensitivity. We compared widths under three matching criteria, none using the empirical null or BH: decoder directions (paper), encoder directions, and functional matching on a held-out probe of 16 fresh views per target plus 2,000 unrelated images, scored by held-out activation correlation. Coverage of the functional probe is now 41/42 targets at 4,096 and 37/42 at 16,384. Entries are dedup-minus-placebo displacement of each target's top latent; brackets divide by that latent's own retraining standard deviation, which controls for baseline instability.

Criterionk = 2: 4,096 / 16,384k = 4: 4,096 / 16,384
Decoder+0.011 [0.05] / +0.170 [1.51]+0.050 [0.66] / +0.408 [3.50]
Encoder-0.002 [-0.14] / +0.104 [1.29]+0.056 [0.68] / +0.277 [3.19]
Functional+0.008 [0.02] / +0.299 [3.48]+0.107 [1.87] / +0.415 [7.03]

The wider dictionary shows larger low-dose displacement under every criterion, including the functional one and after normalising by each latent's own instability. The width effect is therefore not confined to decoder geometry or to the detector's calibration. As in the paper, the ordering reverses at high dose (decoder, k = 32: +0.579 vs +0.346), consistent with greater low-dose susceptibility rather than uniformly larger displacement.

Q3. Natural duplicates. Dose alone does not explain the weaker natural signal: at matched doses k = 2 and k = 4 the planted effect is 3.1x and 4.2x larger (Table 8). The leading difference is similarity. Food-101 contains almost no near-identical pairs, so the clusters at the 0.985 edge threshold are visually similar images rather than re-crops of one photograph, whereas planted copies are crops and mild colour shifts of a single image. We read this as evidence that near-exact repetition, not mere similarity, is what induces a dedicated direction. We will state this scope limitation explicitly. A comparison that varies planted-copy similarity at fixed dose is the natural next experiment, and we will add it if time permits.

All of these analyses will be added to the revised appendix. We hope they address your concerns about calibration, detectability and generality.

Reviewer_cQPBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the stability of learned interpretations, e.g., dictionaries learned by sparse autoencoders. They propose a series of experiments to see how many latents get flagged to help answer the question of whether a direction learned by the sparse autoencoder can recur across retrainings while depending on repeated exposure to particular images.

Critical strengths

Exploring the stability of sparse autoencoders is important for the mechanistic interpretability community.

Critical weaknesses

After running this pdf through GPT-Zero, it estimated the probability of the paper being AI-generated at 87%, and I can see why. There are many statements in this paper that are unnecessarily obfuscating the message of the paper. One of them shows up in my question to the authors, but the whole paper has incomprehensible writing. Some more pointed weaknesses below:

  1. In Section 4.1, you reference that |g| and |D| count activation vectors, but it is not clear what each correspond to in Equation (1)... Is D a set of duplicate latents?
  2. Delta is defined in (2) without an argument, but is used in (3) with an argument called "removal". Not sure how to parse that
  3. The experiments have very specific configurations and it is not clear why, which makes the results inconclusive. For example, why look at the latents that fire at least ten times across a corpus? That might have a reasonable answer, but the paper now reads like an experiment with arbitrarily chosen values, so the results are hard to interpret.
Questions for authors

Your main results section begins with the statement that "At the reference configuration, the removal-minus-placebo displacement gap increases from +0.059 at four extra copies to +0.374 at eight. The targeted win rate is 0.838 [0.771, 0.901], with 37 of 42 targets favouring removal after within-target aggregation."

Why does any of that matter? Please provide an intuitive explanation. The way your results are written, they sound inconclusive at best.

My second question is on Table 7. Why does TopK 64 reduce the latents flagged to 0?

1 comment
AuthorsOpenReview ↗

Response to Reviewer cQPB: intuitive explanation, notation fixes, and TopK 64

Thank you for the careful reading. You are right that the paper is hard to follow; we have rewritten the parts you pointed to. Below we answer each question plainly.

Why the opening numbers matter (intuitive version). Imagine you train an SAE, find a feature, and want to know whether it exists only because one image appeared many times in the SAE's training data. The obvious test is to delete the repeated copies, retrain, and see whether the feature disappears. The problem is that SAE features move even when nothing is deleted: retraining on identical data with a new seed already moves a typical feature by a cosine distance of 0.365. So "the feature disappeared" means nothing on its own.

Our test therefore asks: does the feature move more when its image's copies are deleted than when the same number of random images are deleted? The "gap" is exactly that extra movement.

  • With 4 copies the extra movement is 0.059, about the size of ordinary noise: no dedicated feature.
  • With 8 copies it jumps to 0.374: the feature now depends on the repetition and does not survive its removal.
  • For 37 of the 42 planted images, deleting the copies moved that image's feature more than deleting random images (chance would give about 21).

Why this matters in practice. Practitioners often treat a feature that reappears across retrainings as robust. We show that a feature can be reproducible across seeds and still exist only because one picture was repeated about 5 to 8 times in the SAE corpus. Wider dictionaries need even fewer repeats (about 1 to 2 at width 16,384 to 32,768), and this direction held in all five independent repetitions (Table 4) and again in a new independent experiment run for this rebuttal (zero placebo flags; threshold 11x lower at the wider width). The practical lesson: seed stability is not enough evidence that a feature reflects the model rather than a duplicated image, and SAE corpora should be deduplicated.

Notation in Eq. (1). D is the SAE training corpus, not a set of latents: all cached activation vectors (20,762 images x 49 tokens each). g is one planted group: one original image plus its k near-copies, again counted in activation vectors. rho_jg compares how strongly latent j fires on group g with how strongly it fires on an average corpus vector; rho = 1 means "fires as usual", and we keep latents with rho >= 2. We now define D and g inside Section 4.1.

Delta in (2) vs (3). Agreed, this was sloppy. We now write Delta_j(A): the distance from reference latent j to its closest direction in an SAE retrained under removal arm A, and Delta_j^(r) for the r-th same-data retraining.

Why "fires at least ten times". A latent that almost never fires receives almost no training signal, so its direction is essentially its random initialisation and its distance to any other dictionary is noise. The rule removes only 11 of 4,096 latents, so no result depends on it. The other settings follow standard vision-SAE practice (CLIP ViT-B/16, TopK SAE as in Gao et al. 2025 and Stevens et al. 2025), and each one is varied in the paper: backbone, dataset, depth, sparsity, width and corpus size (Table 6, Appendix Table 26), and the number of candidates per target (Tables 14 and 21). We now state this rationale in Section 3.

Why TopK 64 gives zero flags (Table 7). Two things change at once. The extra movement caused by deleting copies shrinks (0.428 to 0.285), and ordinary retraining noise grows (0.235 to 0.393). The signal becomes smaller than the noise (ratio 1.82 to 0.73), and since the test only flags movement that clearly exceeds a latent's own retraining noise, it can no longer certify any single latent. The effect itself has not vanished: deleting copies still beats random deletion in 70% of comparisons (win rate 0.704), and the gap is +0.385 at k = 64. Zero flags at TopK 64 is therefore a limit of detectability, not evidence that duplication stops mattering. A plausible reading, which we label as interpretation rather than a tested mechanism, is that with 64 active latents per token an image's signal is spread over more latents, and the dictionary is less reproducible across seeds.

On AI assistance. As required by ICLR, the paper's AI-use statement discloses that AI tools assisted with drafting and readability, among other tasks. All experiments were run by us and every number was checked against the run outputs. We take your point that the writing obscured the message; the revision targets exactly that.

Changes in the revised PDF. (1) an overview figure of the design (reference SAE, 40 same-data retrainings, the four removal arms); (2) a plain-language "what the audit measures" paragraph at the start of Section 5, along the lines above; (3) corrected notation for D, g and Delta_j(A); (4) a short rationale for each fixed setting; (5) fewer hedging sentences throughout.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of New Evidence and Scope of Conclusions

Dear Reviewers and Area Chair,

Thank you for your thoughtful and constructive reviews. We appreciate the shared recognition that the reliability of SAE features is an important question for interpretability, and that the study is extensive and well controlled. Below, we summarize the new evidence and clarify the scope of our conclusions.

Central contribution and scope. Our main contribution is an instability-controlled audit that asks whether an SAE direction moves more when a target image's repetitions are removed than when the same number of random images are removed, measured against that direction's own variability across 40 same-data retrainings. Our claims concern the provenance of dedicated directions in an SAE dictionary fitted to a particular corpus, not memorization by the frozen backbone or loss of information, which combinations of retrained directions largely recover. We ran a new, fully independent experiment (new targets, crops, cache and reference dictionaries; widths 4,096 and 16,384; 40 same-data and 10 held-out retrainings each). The original procedure replicates: 110 to 113 flagged latents at width 4,096 and 215 to 230 at 16,384, zero on every placebo and held-out retraining, and a paired width ratio of 11.3 [6.7, 18.3]. The width ratio now exceeds one in all six independent experiments.

Reviewer cQPB: clarity, notation, and fixed settings. Section 5 now opens with a plain-language explanation of what the audit measures: at four extra copies the removal-minus-placebo gap is close to retraining noise, while at eight the target's direction no longer survives, and 37 of 42 targets move more under removal than under placebo. Section 4.1 now defines D and g explicitly and writes the displacement as Delta_j(A) for each removal arm. We state the rationale for each fixed setting; for example, the ten-firing threshold removes only 11 of 4,096 latents whose directions remain near initialisation. We also explain why TopK 64 yields no flags: the removal gap shrinks while retraining variability grows, so the signal falls below the per-latent noise, although the displacement remains positive.

Reviewer oa8M: isolation of targets, candidate sets, and behaviour on new images. Removing one target group at a time reproduces the joint-removal effect on that target's own latents (+0.252 versus +0.254, share 0.99), while latents assigned to the other 41 targets do not move (-0.0001). On the same candidate set, a raw-distance null flags no latents while the per-latent standardised null flags 110 to 230, so detection comes from conditioning on each latent's retraining variability rather than from testing fewer latents. A new held-out probe containing unseen views of each target raises functional coverage from 11/42 and 0/42 targets to 41/42 and 37/42. Under functional matching, the wider dictionary again shows larger low-dose displacement (+0.299 versus +0.008 at k = 2).

Reviewer ZfcS: statistical procedure, width versus detectability, and natural duplicates. A stability-stratified null removes the tail inflation we reported (most-stable tercile P(p <= .01) from 0.0248 to 0.0062) while preserving the discoveries and the width ratio (8.0 [4.7, 14.6]). We also report a limitation: under Benjamini-Yekutieli only the wide dictionary retains discoveries, and under Holm neither width does. Discovery counts are therefore treated as procedure-dependent summaries, and the quantitative claims rest on placebo-adjusted continuous displacement. The low-dose width ordering holds under decoder, encoder, and functional matching, none of which uses the empirical null or BH, and it persists after normalising by each latent's own retraining variability. For natural near-duplicates, the planted effect remains 3.1 to 4.2 times larger at matched doses, indicating that near-exact repetition rather than similarity alone induces a dedicated direction; we state this as a limit on generalisation.

Scope of conclusions. The additional results support duplication dependence of dedicated SAE directions and a replicated width effect in the tested settings. They do not establish a universal duplication threshold or formal false-discovery-rate control, and absolute thresholds still vary across experiments. The revised manuscript adds an overview figure, clarifies notation and settings, and consolidates the new analyses in a new appendix; text added during the discussion period is shown in blue.

We appreciate the reviewers' and Area Chair's consideration and hope that the additional evidence and clarified claims support a reassessment of the submission.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.