- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the energy bias of reconstruction-based feature extractors, such as sparse autoencoders (SAEs), which can prioritize dominant background structure over low-energy target-specific components. To address this limitation, the authors propose SCALPEL, a contrastive SAE framework for fine-grained representation-level unlearning. SCALPEL learns target-selective sparse features by contrasting multiple views of each forget target against retain and Wiki examples. At inference time, selected feature contributions are subtracted from residual-stream activations in an input-dependent manner, without modifying the original model parameters.
Critical strengths
The paper identifies an important limitation of reconstruction-based feature extraction for fine-grained unlearning and provides a theoretical explanation of the energy bias induced by reconstruction objectives.
Critical weaknesses
Insufficient empirical evidence connecting the energy gap to forget-specific information: In Section 3 and Figure 2, the authors report an approximately 329-fold energy gap between the top-three PCA components of background activations and the top-three PCA components of forget activations after removing the dominant background directions. However, these quantities are computed from different distributions and under different transformations. Moreover, the background-orthogonal residual is not necessarily equivalent to forget-specific information, as it may still contain other shared or irrelevant variation. Although Appendix A.3 clarifies that the reported gap does not represent the energy ratio of the complete forget and background manifolds, the evidence remains insufficient to establish that the measured residual energy specifically reflects forget-target information or that the observed gap explains the selectivity limitations of reconstruction-based extractors. The authors also state that the observed gap is substantially larger than that obtained under a random split, but the corresponding numerical results and distributions are not provided. This claim could be strengthened through: (1) quantitative distributions of energy gaps under matched random or pseudo-forget splits using the same analysis pipeline; (2) an intervention-based analysis verifying whether the residual components actually encode forget-specific information; and (3) an analysis relating target-specific energy gaps to the unlearning selectivity of reconstruction-based methods and SCALPEL.
Lack of a Tabulated Comparison at Matched Forgetting Levels: Although the paper presents forgetting–utility trade-off curves in Figure 3 and Appendix F, it does not provide a consolidated numerical comparison of the methods at matched forgetting levels. Such a comparison would make it easier to assess the practical performance differences between SCALPEL and the key baselines, particularly RMU and Gradient Difference. I recommend reporting quantitative results at matched forget-set ground-truth answer probabilities, together with corresponding retain-set utility and Wiki performance. Relevant forgetting metrics, such as ROUGE, truth ratio, and privacy leakage, could also be included at the selected operating points. In addition, evaluating general reasoning capabilities would help determine whether the intervention preserves broader model utility beyond the existing TOFU and Wiki evaluations.
Questions for authors
SCALPEL uses retain and Wiki activations as background examples for contrastive learning and feature selection. How sensitive are the selected features and unlearning performance to the choice of background distribution? In particular, would the method remain effective when replacing Wiki with another dataset or using only the retain set?
Section 4.2 motivates the use of the fourth-to-last layer as a compromise between factual representation and next-token prediction. However, how sensitive is SCALPEL to the intervention layer? Does the same choice consistently provide strong target selectivity and unlearning performance across Qwen, Llama, and Gemma?
Section 4.4 describes the roles of common/difference losses, but their precise formulations and contrastive pair construction remain unclear. Could the authors explicitly define both objectives, including their positive and negative sampling strategies and how the two latent blocks are trained?
Thank you for the time you spent reviewing our paper. Your feedback was very useful. We have decided to withdraw our paper because, after conducting more advanced experiments, we realized that it still needs more work and maturity.
Thank you again for your feedback.