- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies concept erasure in a toy setting with antipodal pairs in superposition. They show that with features represented as antipodal pairs, linear erasure of one feature also erases the other. When fine-tuning with a frozen encoder to suppress one of the features, the authors observe two configurations which they call the mirror and shadow solutions. The authors link this toy model study to an observed failure mode in LLM unlearning.
Critical strengths
The aim of the paper of connecting representation geometry to failure modes in LLM unlearning is interesting.
The paper is honest about its limitations.
Critical weaknesses
-
Even if every claim in the paper was well supported, the extreme simplicity of the setting would still limit the implications and relevance. Most of the results are from a setting even simpler than Toy Models of Superposition with a single perfect antipodal pair or with a frozen encoder.
-
The authors' claim that "Two clusters separate clearly" does not seem to be supported by Figure 2 b, casting doubt on whether "mirror" and "shadow" are meaningful and distinct solutions.
-
The paper seems unpolished and uses terminology like 'phase transition's or 'attractor' too loosely. If these terms apply the authors should provide a clearer justification.
Questions for authors
Even if the writing was substantially improved and my concerns in 2) were addressed I do not think I would recommend acceptance due to the very limited scope with a very simple toy model. I would encourage the authors to withdraw this submission and resubmit, complementing the paper either with strong theoretical results that leverage the simplicity of the setting or with more realistic experiments showing how this applies to real LLM unlearning.
2 comments
Thank you for the response.
I believe the paper is taking steps in the right direction, and some of my concerns have been addressed. However, the paper has changed a lot, having to walk back some of its previous main claims, removing key figures and introducing new theory. As such I believe the paper would benefit from an additional round of peer review. I also am not yet convinced of the significance of the results given the simplicity of the setting. Even if you want to stay in the toy setting, I think going beyond antipodal pairs and including other idealized regular polyhedra arrangements like the ones studied in Toy Models of Superposition would yield a more interesting setting to study theoretically in a re-submission. Anything you can say about implications for real models would also strengthen the paper.
I will increase my score from 1 to 2 but I still do not think the paper is ready to be accepted in its current form.
Thank you for distinguishing the presentation problems from the more substantive concern about scope. Following your suggestion, we have strengthened the theoretical and mechanistic analysis that the controlled setting permits. The central question remains when suppression of a feature in a shared representation can be reversed selectively.
1. We no longer infer two mechanisms or attractors from endpoint clusters.
Re-evaluation shows that all 100 original antipodal checkpoints have positive functional B gain. Both originally labeled groups improve under bias restoration, and both fail the absolute-value negative control. Thus, the fitted partition does not establish two distinct recovery mechanisms.
Section 4 and Appendix C now explain the results through functional gain, activation thresholds, and measured recovery. The endpoint figure displays decoder geometry and functional gain without imposing the two-mechanism interpretation. We have removed the claims of a phase transition and established attractors.
2. The theoretical analysis characterizes what the geometry and objective imply.
Proposition 1 extends the exact antipodal obstruction to non-collinear directions: a linear map that preserves unit direction A and removes unit direction B, whose cosine is ρ, must have operator norm at least (1−ρ2)−1/2. The bound is attainable. This characterizes the amplification cost of separation as the directions approach collinearity, while keeping the result distinct from LEACE’s prediction and distortion guarantees.
Theorem 1 identifies a reconstruction cost for the target’s contribution when the target is independent of retained features. Under a fixed-affine response, the excess retained-output loss is a positive-semidefinite quadratic in that contribution. Components affecting retained predictions decay under the corresponding gradient flow; components in the nullspace need not decay. The assumptions and proof are explicit. We do not present this conditional result as a general theorem about nonlinear Adam training.
3. A paired loss-source intervention tests the proposed attenuation mechanism.
Starting from 30 suppressed antipodal models, we unfreeze the encoder and compare three 5,000-step branches:
The starting norm is 0.2000. In both other-output branches, B’s decoder row and bias remain exactly unchanged. The comparison therefore establishes that B’s own output loss is not necessary for attenuation in this model. Training the retained reconstruction tasks can attenuate B when B remains in their inputs. The corresponding paired reduction in linear-probe AUROC is 0.194, with a 95% seed-bootstrap interval of [0.180, 0.209].
Nonlinear probes still achieve 0.718 AUROC after other-output training. This prevents us from mistaking a small contribution or failed linear probe for complete deletion.
4. The extensions test the restrictions of the original setup.
Section 5 includes 30 unselected learned-encoder models, encoder-update interventions at three initial geometries, and 30 checkpointed models each for depth-2 ReLU and depth-1 GELU. These establish meaningful boundaries: depth-2 bias restoration reaches only 0.538 AUROC, while GELU preserves 0.851 AUROC despite output RMS falling to 0.007. The 1,200-run emergence study also reports all-run denominators.
These experiments develop the original recovery question by identifying when the intervention succeeds, fails, or is unnecessary for preserving ranking. Section 7 and Appendix H distinguish this contribution from prior work already demonstrating reversibility and internal changes in unlearning.
Real-LLM validation remains unperformed, and the synthetic-transformer probability change is not used to establish such transfer. The revised contribution is a controlled account linking shared geometry, selective recovery, and the loss sources that attenuate a target representation. We would appreciate reconsideration on the basis of this theoretical and experimental evidence.