acceptodds
Under review as a conference paper at ICLR 2027

Hidden, Not Deleted: How Networks Suppress Entangled Features

Abstract

Concept erasure methods that operate via linear projection assume that features occupy separable subspaces. We show this assumption fails under dense superposition: when two features are forced into an antipodal pair sharing a single subspace, state-of-the-art linear erasure destroys both, not just the target. Networks trained with gradient descent instead solve this problem non-linearly, but not uniformly: they converge to one of two distinct circuit-level solutions depending on initialization, which we call mirror and shadow solutions. We map this bifurcation as a function of feature entanglement, show it reflects a stable attractor structure rather than an artifact of our setup, and use targeted causal interventions to demonstrate that both solutions leave the erased feature's representation fully intact, recoverable with a single scalar patch. This mirrors a failure mode recently observed empirically in LLM unlearning, where suppression rather than deletion allows forgotten knowledge to resurface; our results offer a mechanistic, causally-validated account of why that failure mode occurs.

Primary area:
interpretability and explainable AI
Keywords:
machine learning, mechanistic interpretability, concept erasure, superposition, causal intervention, representation dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bzXFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies concept erasure in a toy setting with antipodal pairs in superposition. They show that with features represented as antipodal pairs, linear erasure of one feature also erases the other. When fine-tuning with a frozen encoder to suppress one of the features, the authors observe two configurations which they call the mirror and shadow solutions. The authors link this toy model study to an observed failure mode in LLM unlearning.

Critical strengths

The aim of the paper of connecting representation geometry to failure modes in LLM unlearning is interesting.

The paper is honest about its limitations.

Critical weaknesses
  1. Even if every claim in the paper was well supported, the extreme simplicity of the setting would still limit the implications and relevance. Most of the results are from a setting even simpler than Toy Models of Superposition with a single perfect antipodal pair or with a frozen encoder.

  2. The authors' claim that "Two clusters separate clearly" does not seem to be supported by Figure 2 b, casting doubt on whether "mirror" and "shadow" are meaningful and distinct solutions.

  3. The paper seems unpolished and uses terminology like 'phase transition's or 'attractor' too loosely. If these terms apply the authors should provide a clearer justification.

Questions for authors

Even if the writing was substantially improved and my concerns in 2) were addressed I do not think I would recommend acceptance due to the very limited scope with a very simple toy model. I would encourage the authors to withdraw this submission and resubmit, complementing the paper either with strong theoretical results that leverage the simplicity of the setting or with more realistic experiments showing how this applies to real LLM unlearning.

2 comments

Thank you for distinguishing the presentation problems from the more substantive concern about scope. Following your suggestion, we have strengthened the theoretical and mechanistic analysis that the controlled setting permits. The central question remains when suppression of a feature in a shared representation can be reversed selectively.

1. We no longer infer two mechanisms or attractors from endpoint clusters.

Re-evaluation shows that all 100 original antipodal checkpoints have positive functional B gain. Both originally labeled groups improve under bias restoration, and both fail the absolute-value negative control. Thus, the fitted partition does not establish two distinct recovery mechanisms.

Section 4 and Appendix C now explain the results through functional gain, activation thresholds, and measured recovery. The endpoint figure displays decoder geometry and functional gain without imposing the two-mechanism interpretation. We have removed the claims of a phase transition and established attractors.

2. The theoretical analysis characterizes what the geometry and objective imply.

Proposition 1 extends the exact antipodal obstruction to non-collinear directions: a linear map that preserves unit direction A and removes unit direction B, whose cosine is ρ\rho, must have operator norm at least (1−ρ2)−1/2(1-\rho^2)^{-1/2}. The bound is attainable. This characterizes the amplification cost of separation as the directions approach collinearity, while keeping the result distinct from LEACE’s prediction and distortion guarantees.

Theorem 1 identifies a reconstruction cost for the target’s contribution when the target is independent of retained features. Under a fixed-affine response, the excess retained-output loss is a positive-semidefinite quadratic in that contribution. Components affecting retained predictions decay under the corresponding gradient flow; components in the nullspace need not decay. The assumptions and proof are explicit. We do not present this conditional result as a general theorem about nonlinear Adam training.

3. A paired loss-source intervention tests the proposed attenuation mechanism.

Starting from 30 suppressed antipodal models, we unfreeze the encoder and compare three 5,000-step branches:

Training branchMean final B encoder-column norm
Other outputs only; B sampled normally0.0025
Other outputs only; input B set to zero0.2000
B reconstruction-plus-suppression loss only0.1756

The starting norm is 0.2000. In both other-output branches, B’s decoder row and bias remain exactly unchanged. The comparison therefore establishes that B’s own output loss is not necessary for attenuation in this model. Training the retained reconstruction tasks can attenuate B when B remains in their inputs. The corresponding paired reduction in linear-probe AUROC is 0.194, with a 95% seed-bootstrap interval of [0.180, 0.209].

Nonlinear probes still achieve 0.718 AUROC after other-output training. This prevents us from mistaking a small contribution or failed linear probe for complete deletion.

4. The extensions test the restrictions of the original setup.

Section 5 includes 30 unselected learned-encoder models, encoder-update interventions at three initial geometries, and 30 checkpointed models each for depth-2 ReLU and depth-1 GELU. These establish meaningful boundaries: depth-2 bias restoration reaches only 0.538 AUROC, while GELU preserves 0.851 AUROC despite output RMS falling to 0.007. The 1,200-run emergence study also reports all-run denominators.

These experiments develop the original recovery question by identifying when the intervention succeeds, fails, or is unnecessary for preserving ranking. Section 7 and Appendix H distinguish this contribution from prior work already demonstrating reversibility and internal changes in unlearning.

Real-LLM validation remains unperformed, and the synthetic-transformer probability change is not used to establish such transfer. The revised contribution is a controlled account linking shared geometry, selective recovery, and the loss sources that attenuate a target representation. We would appreciate reconsideration on the basis of this theoretical and experimental evidence.

Thank you for the response.

I believe the paper is taking steps in the right direction, and some of my concerns have been addressed. However, the paper has changed a lot, having to walk back some of its previous main claims, removing key figures and introducing new theory. As such I believe the paper would benefit from an additional round of peer review. I also am not yet convinced of the significance of the results given the simplicity of the setting. Even if you want to stay in the toy setting, I think going beyond antipodal pairs and including other idealized regular polyhedra arrangements like the ones studied in Toy Models of Superposition would yield a more interesting setting to study theoretically in a re-submission. Anything you can say about implications for real models would also strengthen the paper.

I will increase my score from 1 to 2 but I still do not think the paper is ready to be accepted in its current form.

Reviewer_3DMXOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a setup where an autoencoder model with n input features has an internal hidden vector with dimension d, where d << n. In this setting the inputs are sparse. The encoder of the system converts an n-dimensional feature vector into the d-dimensiona hidden representation and the decoder reconstructs the sparse n-dimensional feauture vector.

the Toy Model of Superposition (TMS) shows that, when d<<n, multiple feaures (multiple elements of the n-dimensional input) map into the same neuron (or eigen vector) in the d-dimensional space. This concept is refered to as superpostion, and features that map into the same subspace are called superposed features.

Concept erasure refer to the task where the d-dimensional hidden representation is modified so that the decoder does not show certain features in its n-dimensional output vector. When d<<n, erasing a concept from the d-dimensional space with a linear projection will also erase other concepts that were superposed with it.

The entanglement of two features is defined as the cosine difference between the corresponding columns in the weight matrix of encoder, cos(w_a, w_b).

linear removal of one concept (say concept a) projectcs the d-dimensional hidden space into the orthogonal space of w_a. The authors of the paper study a special case, where two features are anti-podal: when the entanglement cos(w_a, w_b) is close to -1. In such a case, the linear removal of concept b will also remove concept a.

The authors then show that feature b can be removed alternatively by training the decoder part of the autoencoder; this approach, unlike linear removal, will not delete feature a from the output n-dimensional feature vector.

the authors then show that the removed feature b can re-appear by increasing its bias term in the decoder.

Critical strengths

None

Critical weaknesses

The writing of the paper is not clear at all. I was not able to understand the abstract or introduction of the paper properly. Concepts are mentioned without prior introduction, which make it challenging for the reader to understand. Here is the very first sentence of the abstract:

"Concept erasure methods that operate via linear projection assume that features occupy separable subspaces. We show this assumption fails under dense superposition: when two features are forced into an antipodal pair sharing a single subspace, state-of-the-art linear erasure destroys both, not just the target." In this text the following are not clear to the reader: "concept erasure", "feature", "separable", dense superposition", "antipodal pair".

The rest of the abstract and introduction follows the same style of writing which, is honestly not readable.

I was not familiar with the toy model of superposition; I tried reading the abstract, introduction, and related work but I could not follow the paper's main studied problem. Only after reading section 3 I could understand the problem at hand.

I could not understand section 4.2, mirror and shadow are not defined. The experiment and its purpose is unclear.

following up, section 4.3 is mostly discussing mirror and shadow, which is not clear.

Questions for authors

Please consider rewriting the paper as it is not clear what the goals and achievements were.

2 comments

Thank you for explaining where the presentation became difficult to follow. We have rewritten the exposition around the same suppression-and-recovery question, introducing the model and task before the terminology.

1. The problem is now stated in ordinary terms.

The introduction begins with a model that reports which attributes are present in an input. We train it to stop reporting B while continuing to report the other attributes. We then ask whether a small change to its readout makes it recognize B again, without falsely reporting B when B is absent.

The central finding is: output suppression can hide a useful feature score, and restoring one saved bias can partially recover detection; allowing the internal representation to change can make that same intervention ineffective.

2. The terms are introduced through the concrete model.

Section 2 defines the input as 16 coordinates, compressed into a four-dimensional hidden vector and then reconstructed:

  • A feature is an input coordinate: zero means absence, and a positive value gives its magnitude.
  • The encoder maps these coordinates into the hidden vector; the decoder reconstructs them.
  • Superposition means that features use overlapping representation directions.
  • An antipodal pair consists of two features represented in opposite directions on the same axis.

Figure 1 illustrates the shared axis, the suppression setup, and measured bias-recovery responses for B-present and B-absent inputs.

Linear concept erasure is introduced in Section 3, after this setup, as an intervention on the hidden representation with a specified prediction-removal objective. We distinguish it from training the output to suppress B. We also correct the original blanket statement about separable subspaces: the geometric result concerns whether a linear map can remove B’s contribution while preserving A’s, rather than claiming that existing erasure guarantees are invalid.

3. “Mirror” and “shadow” are no longer needed to understand the explanation.

We have removed these labels from the main argument. Section 4 instead defines the functional gain: how much increasing input B changes its own output’s pre-activation. All 100 antipodal checkpoints have positive B gain, even though the ReLU output is nearly silent. Restoring the bias exposes some of this useful score.

This explanation also predicts a failure: taking the absolute value of the suppressed score can make A-only inputs produce larger B outputs than B-only inputs. The experiments confirm that failure. Bias restoration, in contrast, increases mean B-presence AUROC from 0.505 to 0.815, with 35.5% true positives at 1.09% false positives.

The paper now follows a single sequence: shared geometry and representation edits (Section 3), selective recovery (Section 4), and the effects of architecture and trainability (Sections 5–6). Captions and appendices are self-contained, and Section 7 places the contribution in the relevant literature.

We would appreciate your reassessment of whether this presentation makes the research question, mechanism, and supported achievements clear.

Reviewer_3DMXOpenReview ↗

changing score

I thank the authors for the major revision, the text reads a lot better now. I am increasing my score to a "weak reject".

Reviewer_5LFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how networks suppress a feature that shares a representation direction with another feature. The problem setting is interesting, and targeted interventions are then used to restore the suppressed output. Additional experiments examine deeper networks, GELU activations, naturally emerging feature geometry, and a small synthetic transformer.

Critical strengths
  1. The paper is generally clear and easy to follow. The distinction between the two proposed solutions is illustrated well, although some conclusions are stronger than the supporting experiments.
  2. I like the problem setting, which connects feature geometry and suppression, which makes their relationship easy to measure.
Critical weaknesses
  1. Although the problem setting is interesting, the experimental setup makes representation retention unavoidable, leaving the main explanation incomplete.
  2. The recovery experiments do not yet distinguish correct feature recovery from simply increasing the target output. In Fig. 5, restoring the bias or taking the absolute value of a pre-activation increases the output for feature B. However, this does not show that the patched model correctly identifies when B is present. For example, if the absolute-value patch also makes B activate on inputs containing only A, the increased readout would not constitute successful recovery. Appendix A notes that this case was not separately checked.
Questions for authors
  1. If you can provide more details on your experiments and make it easy for me to understand, it will be great.
  2. How accurately does the model reconstruct B on inputs where B is present, and how often does it incorrectly activate B when B is absent?
2 comments

Thank you for identifying the missing negative-input test. We have now performed it on all 100 original antipodal checkpoints. It distinguishes a valid, partial recovery result from an intervention that increases output without recovering the correct feature.

1. Recovery is now evaluated on both B-present and B-absent inputs.

Section 4, Tables 2–3, and Figure 2 report evaluations on 4,096 independent sparse inputs, including background features and A/B co-occurrence. No checkpoints are excluded based on recovery performance.

InterventionB-presence AUROCB-present MSEB-absent MSE
Suppressed model0.5050.397<0.000001
Absolute-value patch0.1590.2290.9119
Saved bias restored0.8150.2230.0014

Values are means across the 100 checkpoints; the manuscript also reports seed standard deviations.

For bias restoration, the test true-positive rate is 35.5% at a false-positive rate of 1.09%. Each model’s threshold is determined from the 99th percentile of B-absent scores on a separate 8,192-input validation set. Thus, these rates are measured on held-out inputs rather than obtained by selecting a threshold on the test set.

These results establish partial selective recovery, with substantial remaining reconstruction error. Thirty additional paired antipodal models, using biases saved immediately before suppression, give a consistent AUROC change of 0.504 → 0.820.

2. The absolute-value intervention fails the control you proposed.

On matched A-only and B-only inputs, its B-presence AUROC is zero in every one of the 100 original antipodal checkpoints: it ranks A-only inputs above B-only inputs. Equation (6) explains this reversal. We therefore withdraw the interpretation of this intervention as successful feature recovery.

We also corrected a plotting error in the original Figure 5: its script expanded aggregate summaries into apparent individual curves and points. Those were not measured seed-level trajectories and should not have been presented as such. The replacement recovery plots use actual checkpoint measurements. We apologize for this error.

3. Frozen-encoder retention is now treated as a controlled reference case.

We agree that fixing the encoder preserves its representation by construction. The informative question in that setting is whether a specified intervention restores discrimination without activating the target indiscriminately.

Sections 5–6 test the boundary by allowing representation parameters to change. With full encoder adaptation, bias-restored AUROC falls to 0.501, compared with 0.820 for the paired fixed-encoder models. A separate loss ablation shows that training only the other outputs can reduce B’s mean encoder-column norm from 0.2000 to 0.0025, even when B’s output has no loss and its decoder parameters remain unchanged. Setting input B to zero in the matched branch leaves its encoder column unchanged. Nonlinear probes still detect residual information, so we describe this as attenuation, not complete deletion.

The revised paper consequently separates three measurable outcomes: output suppression, selective readout recovery, and representation attenuation. We would appreciate your reassessment of the recovery claim and soundness in light of these direct controls.

Thanks for your detailed rebuttal. However, I still think that the scope of this paper is not well established and limited. I am happy to keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.