- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies why SAE-based concept erasure can leave target information behind after the identified concept latents are removed. It shows that feature absorption predicts this leakage, as the concept may instead be represented by other SAE latents or the reconstruction residual. Based on this observation, the paper proposes a training-free operator that removes target-aligned components from both channels. Experiments across several SAE variants, Gemma models, and orthographic, syntactic, and semantic concepts show substantially lower probe-based leakage with limited changes to non-target information and general model utility.
Critical strengths
-
This paper identifies feature absorption as a concrete and measurable source of failure in SAE-based concept erasure, supported by consistent token-level and cross-SAE results.
-
The proposed operator is simple and training-free, and the experiments cover multiple SAE training methods, three model scales, and several concept types while showing limited effects on non-target information and model utility.
Critical weaknesses
i) The central relationship between feature absorption and editing failure is partly built into the definitions and evaluation protocol.
A fully absorbed example is required to have inactive parent latents while remaining detectable by the original concept probe. Since standard parent ablation removes only the parent contributions, it leaves the activation of such an example unchanged. Therefore, the near-perfect leakage rate on fully absorbed examples follows directly from their construction, as the paper itself notes in Appendix A.3. Similarly, the absorption fraction and post-edit margin are both calculated using the same probe direction. The strong correlation in Figure 1 therefore does not provide fully independent evidence that absorption predicts editing failure.
The cross-SAE analysis has a related issue. Overall leakage is calculated directly from the proportion of fully absorbed examples and their leakage rate, which is already close to one by construction. Using the proportion of fully absorbed examples to predict overall leakage in Table 9 consequently introduces a direct mathematical dependence between the predictor and the target. The authors should separate this definitional relationship from genuine out-of-sample predictive value by using independently trained probes, holding out entire concept families rather than only SAEs, and adopting leakage measures that are not constructed from the same absorption statistics.
ii) The experiments do not isolate the contribution of the absorption-aware design or independently establish that the target concept has been erased.
The method uses the same fixed probe direction to define absorption, select candidate latents, perform the edit, measure leakage, and construct the recovery attack. Because the operator explicitly removes information along this direction, a low score from the same probe is an expected consequence of the intervention and does not show that the concept is no longer linearly or nonlinearly recoverable. Existing concept-erasure methods instead retrain classifiers after each projection [a], optimize against an adaptive classifier [b], or guarantee that no linear classifier can recover the target concept [c]. Prior probing work also cautions that probe performance alone does not establish how a representation affects model behavior [d]. The paper should therefore evaluate newly trained linear and nonlinear probes and include concept-related generation or downstream behavioral tasks.
The paper also omits a direct baseline that projects the entire edited activation away from the target probe direction. Without this comparison, it is unclear whether the improvement comes from the proposed decomposition across absorbing latents and the reconstruction residual or simply from removing the evaluator’s concept direction. Moreover, the candidate set includes every non-parent latent whose decoder direction exceeds a cosine threshold, rather than only latents independently identified as absorbing the target concept. With the threshold used for Gemma Scope 2, Table 16 shows that approximately 33%–39% of all latents are selected. The threshold also changes across SAE families without a general calibration rule. These results make the operator look more like broad probe-guided projection than a selective correction of absorption channels.
[a] Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. ACL 2020.
[b] Linear Adversarial Concept Erasure. ICML 2022.
[c] LEACE: Perfect Linear Concept Erasure in Closed Form. NeurIPS 2023.
[d] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. TACL 2021.
Questions for authors
-
How does the proposed method perform when concept leakage is evaluated using newly trained linear and nonlinear probes rather than the same fixed probe used to construct the edit?
-
How does the absorption-aware operator compare with a direct baseline that projects the entire edited activation away from the target probe direction, using both leakage and collateral-damage metrics?
-
How should the cosine threshold be selected for a new model or SAE without using evaluation leakage, and does the method remain selective when the candidate set contains more than one-third of all SAE latents?