acceptodds
Under review as a conference paper at ICLR 2027

Principled Nonlinear Concept Erasure and Steering

Abstract

Learned representations often encode sensitive attributes, such as gender or race, that may be undesirable or unsafe to retain. Concept erasure aims to remove this information while preserving task-relevant content. Existing methods remain incomplete: linear methods provide guarantees only against linear probes, while nonlinear methods achieve stronger empirical erasure but lack formal guarantees and often sacrifice utility. In this work, we introduce a principled conditional autoencoder framework for nonlinear concept erasure. We follow the principle that erasure should remove only the sensitive attribute and nothing else, and we show that, under the stated assumptions, perfect reconstruction together with population-level independence is *sufficient* for perfect nonlinear guardedness, with preservation guarantees in the imperfect-reconstruction regime. Empirically, we attain a superior utility-erasure trade-off across multiple datasets, sensitive attributes, and language models, surpassing even the supervised variants of competing methods despite using no target-task labels, while being simpler to optimize and converging substantially faster. Beyond erasure, our framework yields *concept conversion (representation steering)* as a by-product: by changing only the target concept at inference time, it produces representations corresponding to different concept values—at no additional training cost—a capability no prior erasure method provides, and on which we likewise demonstrate strong empirical performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Concept erasure, representation steering, autoencoder

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s9J4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies concept erasure in language model representations. The goal is to remove a sensitive attribute, such as gender or race, and keep other useful information. The authors propose a simple conditional autoencoder. The encoder maps a representation X to an erased representation Z. The decoder rebuilds X from Z and the concept label C. Training uses an MSE reconstruction loss plus an HSIC penalty between Z and C. The authors argue that reconstruction is a better way to keep information than the HSIC-based objective in Obliviator.

On the theory side, the paper shows that good reconstruction plus independence gives perfect nonlinear guardedness. It also gives a bound for the case of imperfect reconstruction. Under a symmetry assumption, the same decoder gives exact concept conversion by changing C. Experiments cover Bias in Bios and two Dial tasks, with representations from BERT, GPT-2, DeepSeek-7B, and LLaMA-1B. The method shows a clearly better utility-erasure trade-off than prior methods, including supervised Obliviator, without task labels.

Critical strengths
  1. I think the method is very simple and easy to use. The objective has only two terms. It is also reported to train faster than prior nonlinear methods (although it is expected).
  2. I think the empirical results are strong. On Bias in Bios with BERT, the method keeps about 74% profession accuracy at near-random gender accuracy. This is much higher than other methods. Similar gains appear across four language models and the Dial tasks.
  3. I like the explanation of why reconstruction works well for preservation. Under independence, maximizing I(X; Z) equals minimizing H(X | Z, C). The paper also explains why maximizing HSIC is different from maximizing mutual information. I think this is a useful insight for this area.
  4. I appreciate that the authors are clear about the scope of the theory (although a little wordy, I am afraid). They state that the guarantees do not cover finite-sample training. They also openly discuss the independence assumption between U and C, and study what happens when it only holds approximately with a controlled simulation.
Critical weaknesses
  1. I think the paper should discuss the fair representation learning literature more directly. A conditional autoencoder with an independence penalty is close to earlier work, based on my knowledge. These methods also rebuild the input from the latent code and the attribute. They can also change the attribute at inference time. I would suggest comparing with at least one of them and softening claims such as “a capability no prior erasure method provides”.

  2. I think the theoretical guarantees are narrower than the abstract suggests. Based on my understanding (please correct me if I am wrong), the erasure result mainly follows from the independence condition itself. There is no bound when HSIC is small and not zero, isn't it? The preservation results are stated given C, for example I(U; Z | C) = H(U). In practice, Z is used without C. Z alone is only guaranteed to be useful under Assumption 2. I think this assumption is hard to satisfy for continuous U. It would help to state these limits in the main claims.

  3. I think the generation steering experiment does not fully match the theory. I think the update rule h′ = h + β[d(e(h), c′) − d(e(h), c)] treats every state as toxic. It also uses β > 1. So Corollary 3 does not directly apply here? I am not 100% sure here. Please correct me here if I am wrong too.

Questions for authors

Please refer to the Weaknesses section; I would appreciate it if the authors could address those points in the rebuttal.

4 comments

We thank the reviewer for the thoughtful review and for recognizing the simplicity of our method, its strong empirical results, and the motivation for using reconstruction rather than HSIC for preservation. We address each concern below.

W1 — Relation to prior autoencoder-based methods.

We acknowledge the earlier work on conditional autoencoders and independence constraints, particularly Shaham et al. (2022), which we already discuss in Section 2. We deliberately build on the same underlying objective, but our contribution goes well beyond applying an existing architecture to another task. Shaham et al. focus on latent-variable recovery and do not establish guarantees for nonlinear concept erasure, complementary-information preservation under imperfect reconstruction, or exact concept conversion. We extend this theoretical framework to establish precisely these guarantees, addressing fundamental questions in nonlinear concept erasure and steering that were not studied in the earlier work.

To our knowledge, ours is the first framework in concept erasure and steering to jointly establish these theoretical guarantees, providing a principled foundation for nonlinear erasure, maximal information preservation, and exact concept conversion. Despite substantial progress in the field, these questions had not been addressed together theoretically.

Our contribution is therefore not a new autoencoder objective, but the theoretical framework and its implications for concept erasure and steering. Since we deliberately adopt the same underlying objective as Shaham et al., a direct empirical comparison would be less informative, as our contribution is not an improvement over their autoencoder objective or architecture. Instead, we compare against existing concept-erasure methods, including Obliviator, KRaM, and FaRM, to evaluate the practical benefits of our framework in the setting it is designed to address. Our statement that this is "a capability no prior erasure method provides" refers specifically to these existing concept-erasure methods, rather than to conditional autoencoders in general.

W2 — Scope of theoretical guarantees.

1. Erasure and preservation. While independence alone guarantees nonlinear erasure, it also permits trivial solutions such as constant representations. Reconstruction is essential to guarantee maximal information preservation alongside erasure, with I(X;Z)=H(X∣C)=H(U)I(X;Z)=H(X\mid C)=H(U) under our assumptions. The contribution is establishing these guarantees jointly, rather than independence alone.

2. Imperfect independence. To keep the manuscript simple and elegant, we originally focused on the main theoretical insights and left out the more technical analysis of imperfect independence. However, given the reviewer's concern, we have now added quantitative erasure guarantees for small but nonzero population HSIC, together with full proofs in the revised Appendix A.5.

The main result is that approximate independence provides approximate protection against nonlinear concept recovery. Specifically, for any bounded-norm RKHS adversary attempting to predict the sensitive concept from the erased representation ZZ, the difference in its expected prediction scores across concept groups is bounded by O(HSIC(Z,C))O(\sqrt{\mathrm{HSIC}(Z,C)}).

Importantly, we extend this result beyond RKHS adversaries to every fixed bounded continuous adversary, including the MLP probes used throughout our experiments. Using the universality of the Gaussian kernel on compact representations, we show that their expected prediction scores become indistinguishable across concept groups as population HSIC approaches zero, directly connecting the theoretical guarantee to our practical evaluation.

Intuitively, the smaller the dependence between ZZ and CC, the less an adversary can distinguish the concept groups from the erased representation. While the convergence rate depends on the adversary's RKHS approximability and the guarantee is not uniform over arbitrary classifiers, it applies to the bounded continuous prediction scores of the nonlinear probes used in practice. Our result thus directly connects the independence term we optimize to quantitative nonlinear concept erasure, extending the theory beyond the ideal setting.

Together, these results strengthen our theoretical guarantees for both ideal and approximate settings, complementing our empirical erasure and steering results, including the additional comparisons presented in our response to ELSp (W3).

W3 — Applicability of Corollary 3 to generation.

We thank the reviewer for this observation. The reviewer is correct that our original intervention assumes a toxic source concept at every generation step, and that β>1\beta>1 is not covered by Corollary 3. To address this, we conducted an additional experiment with a gated intervention that follows Corollary 3 in the ideal regime, without requiring an external classifier.

Relation to the theory. For a state whose true concept is cc and β=1\beta=1, our residual update can be written as

h′=d(e(h),c′)+[h−d(e(h),c)]⏟reconstruction residual.h'=d(e(h),c')+\underbrace{[h-d(e(h),c)]}_{\text{reconstruction residual}}.

Under perfect reconstruction, the residual vanishes, and the update reduces exactly to the conversion map in Corollary 3. In practice, retaining this residual helps preserve information not captured by the decoder. The remaining issue is that the original intervention also modifies states that may already belong to the target concept.

Gated steering. We address this by estimating the source concept of each state using our own trained autoencoder, without any external classifier, as the concept under which the decoder best reconstructs the state:

c^(h)=arg⁡min⁡c∈{toxic, non-toxic}∥h−d(e(h),c)∥22.\hat c(h) = \arg\min_{c \in \{\text{toxic},\, \text{non-toxic}\}} \|h-d(e(h),c)\|_2^2.

We apply the intervention only to states with c^(h)=toxic\hat c(h)=\text{toxic}, leaving all other states unchanged. Under the assumptions of Corollary 3, when the correct concept is uniquely identified by reconstruction, this recovers the true source concept. With β=1\beta=1, the gated intervention therefore implements the exact conversion map in the ideal regime.

Importantly, this introduces no additional decoder evaluations or separate model: the original residual update already computes both d(e(h),toxic)d(e(h),\text{toxic}) and d(e(h),non-toxic)d(e(h),\text{non-toxic}). Gating only compares their reconstruction errors to the original state, adding negligible computational overhead.

Additional experiment. We evaluate this variant using the same Qwen2.5-7B setup and autoencoder as Table 1, on 1,000 toxic and 1,000 non-toxic held-out prompts. The latter allow us to test whether steering unnecessarily changes already non-toxic generations.

MethodToxic EMT ↓Toxic PPL ↓Non-toxic PPL ↓
Untouched0.1633.443.44
Original, β=1\beta=10.0743.813.72
Gated, β=1\beta=10.0743.743.47
Original, β=2\beta=20.0374.775.14
Gated, β=2\beta=20.0434.553.61
Original, β=3\beta=30.0186.868.02
Gated, β=3\beta=30.0275.993.93

The main finding is that the theoretically motivated intervention remains effective while better preserving unaffected generations. At β=1\beta=1, gating achieves the same toxicity reduction as our original intervention (EMT 0.074), with lower perplexity. Moreover, the gate activates on approximately 66–69% of generation steps for toxic prompts, compared with only 26% for non-toxic prompts. Consequently, even at β=3\beta=3, non-toxic perplexity remains at 3.93, compared with 8.02 without gating.

Regarding β>1\beta>1, we agree that Corollary 3 applies to β=1\beta=1. Larger values extrapolate along the learned conversion direction, similarly to the strength parameters used by other steering methods. They are an empirical extension rather than part of the exact-conversion guarantee. Importantly, the results show that the intervention covered by the theory is already effective, without requiring such extrapolation.

We thank the reviewer again and hope these clarifications, additional theoretical results, and new experiments address the reviewer's concerns. We would appreciate it if the reviewer could reconsider their assessment and consider raising their score in light of these additions. We are happy to clarify any remaining questions.

Thanks for the rebuttal. I appreciate the additional experiments provided as well as the response " Applicability of Corollary 3 to generation". I would like to hear the response from other reviewers and now I keep my positive score.

Thank you for your positive response and for taking the time to consider our rebuttal and additional experiments. We appreciate your thoughtful feedback throughout the review process and are glad that you found our clarifications regarding the applicability of Corollary 3 and the additional experimental evidence useful.

Reviewer_N5wMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an unsupervised conditional autoencoder to erase discrete sensitive attributes from frozen language model embeddings while enabling inference-time representation steering. The paper proposes to use input reconstruction as a way to preserve content as opposed to kernel based similarity approaches.

Critical strengths
  1. Theoretical results based on showing that ideal reconstruction plus statistical independence blocks any nonlinear classifier from recovering the attribute while enabling clean attribute conversion.
  2. Simple algorithm: Replaces unstable adversarial games and slow matrix-eigenvalue steps with one straightforward, single-stage loss -- its bottleneck cannot be mathematically reversed by an attacker to recover the hidden attribute..
Critical weaknesses
  1. Unrealistic independence assumption: The theory assumes sensitive attributes and useful content never correlate, which fails in real text (e.g., topic alone predicts toxicity 71% of the time).
Questions for authors

None beyond what was highlighted in the weaknesses.

3 comments

We thank the reviewer for the positive assessment, and for recognizing both our theoretical results, showing that ideal reconstruction and independence block any nonlinear classifier from recovering the attribute while enabling clean attribute conversion, and the simplicity of our single-stage objective, which replaces unstable adversarial games and slow eigenvalue steps. We address the reviewer's concern below.

W1 — Independence between the concept and useful information.

We agree that independence between the concept and the underlying content is an idealized assumption that does not necessarily hold in real-world data. However, our method does not require this assumption to work in practice. In fact, we already investigate violations of this assumption in Appendix B, where we gradually increase the dependence between the concept and complementary information. The results show that erasure remains near chance throughout, while concept-independent information is preserved. The degradation is gradual and concentrated in information correlated with the concept, with steering remaining effective even under substantial dependence. Our strong empirical results on real language-model representations further demonstrate the method's effectiveness beyond the idealized theoretical setting.

To further address this concern, we conducted an additional experiment on a real-world dataset, explicitly controlling the correlation between the sensitive concept and the downstream task.

Experimental setup. We use DIAL-Mention. We subsample the training data so that the task and concept labels agree in a fraction rr of the examples, with balanced marginals. In other words, knowing the concept label allows predicting the task label with accuracy rr: at r=0.5r=0.5, the labels are independent, while at r=0.8r=0.8, the concept predicts the task correctly in 80% of examples. We consider r∈{0.5,0.6,0.7,0.8}r \in \{0.5, 0.6, 0.7, 0.8\}. All training sets contain 82,770 examples, and we use identical hyperparameters across all settings, with 5 seeds. We evaluate on two test sets: the standard balanced test set, and a test set that preserves the training correlation rr.

We report the changes in task accuracy and race leakage relative to the independent setting (r=0.5r=0.5), in percentage points.

Correlation rrBalanced: Δ task acc. ↑Balanced: Δ leakage ↓Correlated: Δ task acc. ↑Correlated: Δ leakage ↓
0.50.00.00.00.0
0.6−0.4+0.2−0.5+0.5
0.7−1.2+0.4−1.6+1.1
0.8−2.3+0.3−2.7+1.4

The results show that erasure remains stable even as dependence increases. On the balanced test set, race leakage changes by at most 0.4 points, while task accuracy drops by only 2.3 points. Notably, frozen BERT also loses 3.0 points without any erasure, suggesting that the degradation is not primarily caused by our method.

On the correlated test set, frozen BERT benefits from the correlation, with task accuracy rising from 75.8% to 81.1%, consistent with exploiting race information to predict the task. In contrast, our task accuracy decreases by only 2.7 points, consistent with our method substantially reducing the race information that could otherwise be exploited to predict the task. Crucially, race leakage remains at 81–86% for frozen BERT, compared with only 55–57% for ours (50% chance), increasing by just 1.4 points for our method even at the strongest correlation.

Together with our existing Appendix B experiments, these findings provide further evidence that our method remains effective beyond the independence assumption, with limited utility degradation even under substantial task–concept correlation. We have added these results to revised Appendix B.5, clarifying the distinction between our theoretical assumptions and empirical performance.

We thank the reviewer again for the constructive feedback. We hope our responses and the additional experiments have addressed the reviewer's concerns, and we would be grateful if the reviewer would consider raising their score. We are happy to clarify any remaining questions.

Reviewer_N5wMOpenReview ↗

Response to author rebuttal

Thank you for the note, I will keep my score as is noting the lower confidence associated with my review.

Thank you for your response and for maintaining your positive assessment. We appreciate your thoughtful consideration of our work and the opportunity to address your concern regarding the independence assumption through additional experiments.

Reviewer_ELSpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a conditional autoencoder for nonlinear concept erasure and representation steering. The encoder maps the original representation to a latent representation, while the decoder reconstructs the input using the latent representation and the sensitive concept label. Training combines an L2 reconstruction loss to preserve information with an HSIC penalty to remove dependence on the sensitive concept. The paper also studies the theoretical properties of this formulation. Under perfect reconstruction and statistical independence, it shows that the learned representation preserves the maximum possible information about the original input. It further provides an information-preservation bound based on reconstruction error and uses the conditional decoder for concept steering.

Critical strengths
  • The proposed objective is straightforward, with only reconstruction and independence terms. It avoids the iterative training and more complicated optimization procedures of some existing nonlinear erasure methods.
  • The method achieves favorable utility–erasure trade-offs across several datasets and language-model representations. In some settings, it outperforms supervised baselines without using downstream labels in the training loss.
  • The paper is well written, and the proposed method is easy to follow. The additional nonlinear probes, HSIC independence tests, and hyperparameter sensitivity studies provide useful evidence supporting the concept-erasure results.
Critical weaknesses

w1) I find the theoretical contribution rather limited. In the perfect regime, the paper assumes that the learned representation is statistically independent of the sensitive concept and that the decoder can perfectly reconstruct the original representation. Once these two properties hold, maximal information preservation follows directly from standard information-theoretic identities. I think it is useful to include this result because it makes the paper self-contained. However, I do not see a substantial new theoretical contribution in this implication. The imperfect-regime analysis is also relatively straightforward. More importantly, these results are not specific to representations learned using the proposed reconstruction objective. Any representation and decoder satisfying the same reconstruction-error assumptions would admit the same bound. The imperfect-regime analysis also does not provide an erasure guarantee when the learned representation has nonzero dependence on the sensitive concept. Overall, the theory appears to formalize useful properties of the desired representation, but I am unclear about its novelty beyond existing information-theoretic and independence-constrained representation-learning results.

w2) The theoretical argument against HSIC-based preservation does not explain the empirical performance gap. Section 4.1 uses Proposition 3 to motivate reconstruction over HSIC-based preservation, particularly in comparison with Obliviator. However, I do not think the theoretical analysis supports the proposed explanation for the empirical performance gap. Proposition 3 uses a counterexample in which rescaling the representation changes HSIC while preserving mutual information. However, this construction assumes a fixed Gaussian kernel bandwidth. The manuscript itself acknowledges that Obliviator employs normalization, which addresses this scaling issue. Furthermore, when the bandwidth is recomputed using the median heuristic, uniform rescaling leaves the RBF kernel matrix unchanged. Therefore, although Proposition 3 is valid under its assumptions, its counterexample does not directly establish a limitation of the practical baseline. More importantly, the authors acknowledge that HSIC maximization with suitable normalization or constraints can achieve the same ideal information-preserving optimum as reconstruction, under additional assumptions. They then argue that Obliviator does not enforce the required covariance whitening or spectral isotropy. However, the experiments are conducted in the imperfect regime, where neither method is guaranteed to reach its ideal optimum. Even if a particular normalization provides stronger guarantees at the global optimum, this does not imply that it will achieve better results under finite, non-convex optimization. A different normalization may perform better in practice despite having weaker theoretical guarantees. The paper does not provide evidence that enforcing the proposed whitening or spectral constraints would improve Obliviator's empirical performance. Therefore, while the experiments show an advantage for reconstruction-based preservation, the theoretical analysis does not establish that the claimed limitation of HSIC-based preservation is responsible for this advantage.

w3) Insufficient experimental evidence for the advantage of nonlinear steering. I find the steering evaluation less convincing than the concept-erasure experiments. The paper provides several steering results involving gender and profession representations, but much of this evaluation relies on classification-based measurements. The toxicity experiment on Qwen2.5-7B is a useful addition because it evaluates actual text generation. However, I think the experimental setup is still limited. The comparison includes mean matching, activation addition, and MiMiC, but omits more recent steering methods. For example, AcT (ICLR 2025) and LinEAS (NeurIPS 2025) are relevant comparisons. I would like to see stronger comparisons with recent methods and additional generation-based experiments beyond the current gender-related and toxicity settings. Furthermore, the toxicity improvements over MiMiC are relatively small and not consistently favorable to the proposed method. The proposed method achieves better perplexity at several operating points, but it is difficult to establish a clear overall advantage from the current comparisons. Overall, the experiments demonstrate that the method can steer model behavior, but they do not provide sufficient evidence that the proposed nonlinear approach offers a consistent advantage over existing steering methods.

w4) Appendix C.2 states that each configuration is trained with five random seeds, but only the curve with the best utility-minus-leakage score is reported. Selecting the best-performing seed does not show the consistency of the method. Reporting the mean performance and variability across the five seeds would provide a more reliable picture of the results.

Questions for authors

please refer to weakness section.

7 comments

We thank the reviewer for the detailed review, and for recognizing the simplicity of our objective, our favorable utility–erasure trade-offs without downstream labels, and the clarity of the paper and its supporting experiments. We address each concern below.

W1 — Theoretical contribution.

We appreciate the reviewer's perspective, but we believe the simplicity and elegance of the derivation are among its strengths. The main theoretical insight is recognizing that reconstruction and independence together provide sufficient conditions for optimal nonlinear concept erasure, a connection that, to our knowledge, had not been established in the concept-erasure literature despite substantial progress in the field.

While the proof relies on standard information-theoretic identities, identifying the right conditions and showing how they jointly guarantee both nonlinear guardedness and maximal information preservation is the central contribution. These guarantees are independent of any particular architecture or optimization procedure, making them broadly applicable, while our objective provides a simple way to encourage them in practice. Our imperfect-regime analysis further quantifies information preservation under reconstruction error.

To keep the manuscript simple and focused on the main theoretical insights, we originally left out the more technical analysis of imperfect independence. However, given the reviewer's concern, we have now added quantitative erasure guarantees for small but nonzero population HSIC, together with full proofs in the revised Appendix A.5.

Specifically, for any adversary in an RKHS ball, the gap between its expected prediction scores across concept groups is at most O(HSIC(Z,C))O(\sqrt{\mathrm{HSIC}(Z,C)}). We further extend this to every bounded continuous adversary, i.e., any continuous classifier with bounded outputs such as predicted probabilities, including the MLP probes used in our experiments: its ability to distinguish the concept groups vanishes as HSIC approaches zero, at a rate determined by its smoothness. This form is the strongest possible, since no HSIC-based guarantee can hold uniformly over arbitrarily sharp classifiers. Our result thus directly connects the independence term we optimize to approximate nonlinear erasure, extending the theory beyond the ideal regime.

We believe the value of the theory lies in providing a simple, general, and principled foundation for nonlinear concept erasure, rather than in the complexity of the proofs themselves.

W2 — HSIC-based preservation and the empirical gap.

We agree that a fixed-bandwidth scaling example does not settle the issue: normalization and the median heuristic remove uniform scaling. But the difference between HSIC and mutual information goes beyond scale. Mutual information measures how much can be recovered from a representation; HSIC also depends on kernel geometry.

We therefore added a stronger counterexample (Proposition 4, revised Appendix A.1) that works for any bandwidth rule based only on the representation's distribution, including the median heuristic. For XX uniform on nn points, a lossless representation has I(X;Z)=log⁡nI(X;Z)=\log n but HSIC below 1/n1/n. A one-bit summary has only I(X;Z)=log⁡2I(X;Z)=\log 2, yet HSIC remains bounded away from zero. Thus, HSIC can favor keeping one bit over keeping all the information, while reconstruction favors the lossless representation.

Of course, a theoretical counterexample does not explain an empirical performance gap on its own. We therefore ran a preservation-term ablation on Bias in Bios, matching the independence penalty, code size, training budget, and other settings. We compared reconstruction against encoder-level HSIC, normalized HSIC (CKA-type), and an autoencoder with HSIC(x^,x)\mathrm{HSIC}(\hat x,x) replacing reconstruction MSE. Results are means over five seeds, reported relative to reconstruction:

Preservation termTask accuracy change ↑Gender leakage change ↓Train time ↓
Reconstruction (ours)0.00.057 min (1.0×)
HSIC(z,xz,x), encoder−7.0−4.5170 min (2.9×)
Normalized HSIC(z,xz,x), encoder−6.3−4.1191 min (3.3×)
HSIC(x^,x\hat x,x), autoencoder−6.4−5.0210 min (3.7×)

Reconstruction preserves 6.3–7.0 more points of profession accuracy, though with 4.1–5.0 points more gender leakage, and trains 2.9–3.7× faster. Importantly, the last variant retains the autoencoder architecture and changes the preservation loss alone; it still loses 6.4 points of task accuracy. This supports the value of reconstruction itself, rather than attributing the difference only to architectural choices or Obliviator's other training steps.

W3 — Additional generation-based steering experiments and comparisons.

We thank the reviewer for this suggestion. To address this concern, we conducted additional generation-based experiments on Gemma-2-2B, including a new sentiment-steering task and stronger toxicity comparisons against recent methods such as Mean-AcT, Linear-AcT (ICLR 2025), LinEAS (NeurIPS 2025), and CHaRS (ICML 2026), alongside MiMiC and other baselines.

Sentiment steering — Gemma-2-2B

To extend the evaluation beyond gender and toxicity, we introduce a new generation-based sentiment experiment. We use SST-2 sentences for fitting and negative IMDB review prefixes for evaluation, measuring the percentage of generated continuations classified as negative. All methods use the same generation settings, held-out validation protocol, and 5-seed evaluation.

MethodNegative continuations ↓Wiki PPL ↓
Unsteered54.46 ± 1.9424.79
Mean-AcT29.14 ± 1.1125.58
Linear-AcT28.80 ± 1.5125.71
LinEAS24.08 ± 1.4125.64
CHaRS28.56 ± 0.9125.57
MiMiC43.96 ± 1.5025.24
Ours14.68 ± 1.2925.95

Our method achieves substantially stronger sentiment steering, reducing negative continuations to 14.68%, compared with 24.08% for the strongest competing method, LinEAS, and 28.56% for CHaRS, with a modest increase in Wikipedia perplexity.

Toxicity steering — Gemma-2-2B

We also substantially expand the toxicity evaluation by following the original Linear-AcT/LinEAS protocol, using the authors' released ml-lineas codebase. All methods use the same fitting data (1,024 Jigsaw examples per class), generation settings, and evaluation metrics; LinEAS is trained using the 10k optimization steps of its high-data recipe. Layers and intervention strengths are selected on held-out validation data using the published Wikipedia perplexity criterion. Results are averaged over 5 seeds.

MethodRTP Tox. ↓TET Tox. ↓Wiki PPL ↓MMLU ↑
Unsteered4.30 ± 1.0717.79 ± 0.9024.7953.10
Mean-AcT0.56 ± 0.158.11 ± 0.4625.0451.87
Linear-AcT0.54 ± 0.297.61 ± 0.5425.1251.50
LinEAS0.62 ± 0.476.03 ± 0.5126.0450.15
CHaRS0.64 ± 0.238.28 ± 0.5824.9651.79
MiMiC2.50 ± 0.4913.53 ± 0.7724.9152.29
Ours0.36 ± 0.214.03 ± 0.1525.3352.27

Our method achieves the lowest toxicity on both benchmarks: 0.36% on RTP (versus 0.54% for Linear-AcT) and 4.03% on TET, compared with 6.03% for LinEAS and 7.61% for Linear-AcT. Importantly, MMLU remains within one point of the unsteered model (52.27 vs. 53.10), while Wikipedia perplexity stays within the selection budget.

Our intervention. In both experiments, we use the state-gated update described in our response to Weakness 3 (W3) of Reviewer s9J4. The intervention is applied only to states that our autoencoder identifies as carrying the source concept, without requiring any external classifier, additional training, or labels. The gate also requires no additional decoder forward passes, as it only compares the reconstruction errors of the two concept-conditioned decodings already computed by the original update.

Therefore, the new results show that our method provides consistently stronger steering than recent state-of-the-art methods, on both sentiment and toxicity, while preserving fluency and general capabilities.

W4 — Mean reporting and variability across random seeds.

We have updated all concept-erasure results to report the mean across independent training seeds, for both our method and all baselines. Our steering results already reported averages across evaluation seeds, also for all baselines.

We have also added Appendix D.5 (Variability and Uncertainty), summarizing standard deviations across training seeds for concept erasure and standard errors across evaluation prompts for toxicity steering. For our erasure method, task-accuracy standard deviations range from 0.5–1.4 percentage points, and concept-leakage standard deviations from 0.3–2.6 points. The corresponding ranges for Obliviator are 0.2–2.3 and 0.4–3.5 points, respectively.

Finally, we have updated Appendix C.2 (Experimental Setup) to clarify the implementation details and evaluation protocol used in our experiments. These additions provide a more complete and transparent account of our experimental results.

We thank the reviewer again for the thorough and constructive review. We hope our responses and the additional theoretical and empirical results have addressed the reviewer's concerns, and we would appreciate it if the reviewer could reconsider their assessment and consider raising their score in light of these additions. We are happy to clarify any remaining questions.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer,

Thank you again for your detailed review. In response to your concerns, we provided additional theoretical guarantees, stronger counterexamples and controlled ablations, new generation-based steering comparisons against recent methods, and more comprehensive variability reporting.

We believe these additions address your main concerns and further support our original contributions. We would appreciate your feedback on whether your concerns have been resolved or if any questions remain.

Thank you for your time and consideration.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Results

Dear Reviewers and Area Chairs,

We thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the simplicity of our approach, its strong empirical utility–erasure trade-offs across datasets and language models, and the value of studying nonlinear concept erasure and steering within a principled framework.

In response to the reviews, we conducted additional theoretical analyses and targeted experiments to address the reviewers' concerns and further substantiate our original contributions.

On the theoretical side, we established quantitative erasure guarantees for approximate independence (nonzero population HSIC), extending our analysis beyond the ideal setting. We also developed a stronger theoretical counterexample illustrating the limitations of HSIC-based information preservation, clarified our relationship to earlier conditional autoencoder methods, and refined the scope of our guarantees.

On the empirical side, we conducted controlled experiments examining violations of the independence assumption, performed preservation-objective ablations, and provided additional generation-based steering evaluations on sentiment and toxicity against recent methods, including Linear-AcT (ICLR 2025), LinEAS (NeurIPS 2025), and CHaRS (ICML 2026). Our method achieves the strongest steering effectiveness among the compared methods on both new generation tasks, while maintaining strong language-model fluency and largely preserving general capabilities. We also evaluated a theoretically motivated gated intervention that improves preservation of unaffected generations, and provided additional analyses of variability across independent runs.

Together, these additional analyses and experiments reinforce the theoretical and empirical findings of our original submission, providing further evidence for our unified framework for nonlinear concept erasure, information preservation, and steering.

The additional results, theoretical analyses, and clarifications are presented in our detailed responses to the individual reviewers below, where we address each concern separately.

We hope these additions help clarify the contributions of our work and address the remaining concerns. We would be grateful for the reviewers' consideration of these additional results when reassessing the paper, and we welcome any further questions or feedback.

Thank you again for your time and thoughtful engagement.

Area_Chair_vb6DOpenReview ↗

[Open Discussion] Thank you for your review and please check authors' rebuttal

Dear All Reviewers,

We have received mixed opinions on this paper. Thank you all for your thoughtful reviews and constructive feedback. Currently, it is still in the open discussion phase, so reviewers are welcome to have further discussions as well as the authors.

Could you please review the authors' rebuttal and share your updated opinions on whether your concerns have been adequately addressed? Please also let us know if you have any remaining questions or concerns that require further clarification.

Dear Reviewer N5wM,

You provided a positive assessment of this paper. Could you please confirm whether the authors' rebuttal has adequately addressed your remaining concerns, including the independence assumption between sensitive attributes and useful information?

Dear Reviewer ELSp,

You expressed reservations about this paper. Could you please check whether the authors have provided sufficient evidence and clarification to address your concerns, such as the novelty of the theoretical contributions, the justification for reconstruction-based preservation, and comparisons with recent steering methods?

Dear Reviewer s9J4,

Thank you for your follow-up and continued positive assessment. We appreciate your engagement with the authors' rebuttal and welcome any further comments following the discussion with other reviewers.

Thank you all again for your time and valuable contributions to the review process.

Best regards,
AC

Thanks for the message. I have read the authors’ responses to the other reviewers, and I would like to keep my positive score. As someone working in this direction, I find the theoretical contribution particularly valuable. To my knowledge, this is the first framework in concept erasure and steering that jointly establishes guarantees for nonlinear erasure, maximal information preservation, and exact concept conversion. I believe this theoretical exploration will benefit the community, and I encourage the authors to incorporate the clarifications from the rebuttal into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.