- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 1(−1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether sparse autoencoder directions that recur across training seeds depend on repeated images in the SAE training corpus. The proposed audit compares decoder-direction changes after removing duplicate groups or only their extra copies against per-latent retraining variability and size-matched random removals. Experiments on frozen vision-model activations report greater sensitivity at low duplication doses in wider dictionaries and weaker effects for naturally occurring duplicates.
Critical strengths
- The paper addresses an important question about the reliability of SAE features.
- The experimental study is extensive and includes useful controls.
- The authors clearly discuss the limitations of their findings.
Critical weaknesses
- The paper lacks an overview figure illustrating the design, including the removal conditions and retraining comparisons.
- The paper assigns candidate latents to groups based on duplicate-specificity, then measures how their decoder directions change after removal and retraining. However, all 42 groups are modified together, so a latent associated with one target could shift because repetitions of other images were removed. The experiment therefore does not isolate whether the assigned target’s repetitions are responsible for that latent’s change.
- The proposed method accounts for each feature’s usual variation across retrainings when assessing the effect of duplicate removal. However, the baseline comparison also changes which features are tested, with the proposed method using a smaller, more selective set. IMO, a comparison using the same candidates is needed since testing fewer features can itself make detections easier.
- The main experiments track changes in decoder directions after duplicate removal, but this does not directly show how features respond to new images. The functional test has insufficient coverage to compare dictionary widths, while the recovery analysis is limited to near-duplicate variants of the same original image. A broader held-out evaluation is needed to determine whether the geometric width trend is accompanied by changes in feature behavior.
Questions for authors
Refer to weakness section.
Response to Reviewer oa8M
Thank you for the constructive review. Each weakness asked for something we could test, so we ran a new, fully independent experiment: new targets, new crops, a rebuilt activation cache, and a new reference dictionary with its own 40 same-data retrainings (20,762 images, widths 4,096 and 16,384). The paper's procedure replicates on it: 110 to 113 flags at width 4,096 and 215 to 230 at 16,384, zero on every placebo, zero on 10 held-out null retrainings, and a paired width ratio of 11.3 [6.7, 18.3].
1. Overview figure. Agreed. The revised PDF adds a design figure: frozen backbone, cached activations, reference SAE, 40 same-data retrainings, the four removal arms with their placebos, and the comparison of each latent's displacement with its own retraining variability.
2. All 42 groups are removed together. We now remove one target group at a time: 8 targets (two at each k in {8, 16, 32, 64}), two seeds each, each with a size-matched random-removal control (32 new SAEs at width 4,096). In each run we measure the removal-minus-control displacement of the latents assigned to the removed target, and of the latents assigned to the other 41 targets.
Per target the single and joint gaps agree closely (for example +0.489 and +0.474 vs +0.476 for one k = 32 target). One k = 8 target, below this experiment's threshold, moves in neither condition. A latent's displacement is therefore explained by removing its own target's repetitions, not other groups'. One caveat: with about 10 affected latents among 403 tests, a single-group run does not reach the BH threshold, so this evidence is the continuous displacement rather than flags.
3. Comparison on the same candidates. On the same 403 candidates, we replaced the per-latent standardised null with a pooled raw-distance null (no per-latent conditioning). It flags 0 latents in every removal run at both widths, versus 110 to 113 (4,096) and 215 to 230 (16,384) with the paper's null, while both flag zero on placebos. With the candidate set held fixed, detection comes from conditioning each latent on its own retraining variability, not from testing fewer latents.
4. How features respond to new images. We built a held-out probe the SAEs never saw: 16 fresh near-duplicate views of each target, plus 2,000 unrelated Food-101 images outside the corpus. Each target's top reference latent is matched to the retrained latent whose activations correlate best on half the probe; we then evaluate that fixed match on the other half and report 1 minus the held-out correlation. Coverage is now 41/42 targets at width 4,096 and 37/42 at 16,384 (previously 11/42 and 0/42). No null and no FDR are involved.
The functional change tracks the geometric one: it rises with dose, and at low doses it is much larger in the wider dictionary. The geometric width trend is therefore accompanied by a change in how the features respond to held-out images.
All four analyses will be added to the revised appendix. We hope these results address your concerns and would be grateful if you would consider them in your assessment.