acceptodds
Under review as a conference paper at ICLR 2027

Erase It Where You Found It: Matched-Position Erasure Lowers Demographic Dependence at a Measured Cost

Abstract

Linear concept erasure removes a direction from a model’s hidden states. LEACE, the standard eraser, guarantees that no linear probe can recover the concept from the edited states. This paper shows that the token positions at which a direction is removed decide what the erasure removes and what it costs. The demographic direction comes from swapping the demographic term of a bias item and measuring how much the answer depends on it. That direction is then erased only at the term’s tokens, in every layer. At those positions the erasure lowers the measured dependence by 11 to 28 percent on four open models. LEACE fitted to the same swap data, and random subspaces, remove nothing at those positions: a probe guarantee is not a guarantee about the answer. An additive steering vector at the same positions does as well as the erasure. At the last prompt token neither edit removes the dependence, and at every token both harm the model. On new BBQ items the erasure removes the demographic term’s contribution to the answer, whether that contribution is a stereotype or the information the question needs. On items in which the term is the answer, accuracy falls from 0.80 to 0.16 on Gemma-2-2B. The evidence spans four models, three benchmarks, a pre-registered replication on new items, and controls for direction, rank, energy, readout and operator. The site of the edit decides what is removed and what is lost. An erasure result should state where the edit acts and what it costs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
benchmark validity, AI governance, evaluation evidence, construct validity, perturbation audits, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uG1aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that in linear concept erasure, the token positions where a projection is applied largely decide what gets removed and what it costs. The authors measure a model’s dependence on a demographic term by patching activations between two versions of a bias item that differ only in that term. They then erase the resulting direction only at the term’s own tokens. Across four open models and a pre-registered replication on new BBQ items, this matched-position edit reduces the measured dependence, while swap-side LEACE and random subspaces do not, and applying the same edit at other positions is often ineffective or harmful. The authors also show a clear cost: the edit removes the demographic information even when the question needs it, which lowers accuracy.

Critical strengths
  • The paper raises an interesting and practically relevant point: the position where an erasure is applied matters, which is something that existing erasure methods tend to leave implicit.
  • The authors evaluate their method on several open models and benchmarks.
  • The paper includes a fairly detailed appendix and releases code, which helps with reproducibility.
Critical weaknesses

I should mention that I am not an expert in this area, so please take the following points as my impressions and questions rather than firm conclusions.

  1. I found the main effect a bit hard to interpret because it does not seem fully consistent. In the pooled audit, the reduction is not significant on Qwen2.5-7B, and if I read Table 9 correctly, only about half of the pairs show a lower dependence after the edit, so the average effect may come from a smaller group of items. In the pre-registered replication, the main hypothesis does not seem to hold on Gemma-2-2B, and on Llama-3.1-8B some of the tests are marked as limited by the authors. The Llama-3.1-8B result also seems to change with the scoring method (Table 21), and the accuracy gain on Gemma-2-2B in the pooled audit does not appear in the replication. I also noticed that the first token of the gold option often starts other options as well (Table 7), which might make the pooled audit’s measure a bit noisy. Could the authors comment on how robust they think the effect is?

  2. I was a little unsure about the LEACE comparison. As far as I understand, the main LEACE baseline uses a label that mixes many different term pairs, which the authors themselves note is not one coherent concept, so its weak result might be partly expected. In Table 25, when LEACE or the mean difference is fitted on a coherent gender label, both seem to reduce the dependence, and on Gemma-2-2B the mean difference even appears to do slightly better than the proposed erasure. The erasure also seems to remove much more activation energy than LEACE in that table, which makes the “about twice as much” comparison a bit hard for me to read. It might help to compare against these stronger baselines at matched energy. Would the main conclusion about LEACE still hold in that setting?

  3. Since the method needs to know where the demographic term appears in the prompt, I wondered whether a simpler approach, like replacing the term with a neutral word such as “person,” might work similarly. The results seem to suggest that the erasure behaves somewhat like hiding the term from the model, since accuracy drops a lot on items where the term is actually the answer (Table 15). A comparison with such prompt-level changes would help me understand the benefit of the activation-level edit. What do the authors see as the main advantage of their approach over simply rewriting the prompt?

  4. The experiments use models with at most eight billion parameters, and the replication uses two of them. The study also focuses on one attribute family and needs the attribute to appear as explicit tokens, which the authors also mention as a limitation. I am curious whether the findings would carry over to larger or newer models, or to other types of concepts. Have the authors tried any such settings?

  5. The presentation of the paper needs substantial improvement. The main text is very dense, relies on many custom terms (such as pooled audit, fresh replication, sequential source state, and calibrated cells), and reports almost all results through long tables packed with numbers. There is only one figure in the main text, so the key findings are very hard to grasp at a glance. The figures that do exist are also difficult to read: the text and legends are small and unclear, and in Figure 4 some data markers overlap with their text labels. Overall, the paper reads as if it was prepared in a hurry, and it would need a clearer structure, fewer custom terms, and well-designed figures summarizing the main results to be accessible to readers.

Questions for authors

See the weaknesses.

Reviewer_dfbYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that the token positions at which a linear concept-erasure direction is applied determine what the erasure removes and what it costs. Demographic dependence is measured by activation patching: two variants of a bias item differ only in a demographic term, and the term's residual states are swapped across variants to see how much the answer moves. The erasure direction is the leading SVD direction of the same swap's activation differences, and it is removed only at the term's token positions across all layers ("matched-position edit"). This is compared against LEACE, mean-difference and random bases, against other application sites (last token, every position, a non-demographic control span), and against additive steering vectors. Evaluation consists of a pooled audit on four models and three bias benchmarks, and a pre-registered replication on new BBQ items with two models. The authors report an 11–28% reduction in the patching effect at the matched positions, no reduction from LEACE or random bases, model damage when the direction is applied everywhere, and an accuracy cost on items where the demographic term is the correct answer (e.g., 0.80 → 0.16 on Gemma-2-2B).

Critical strengths
  1. Rigorous and transparent evaluation design (Sections 3.4, 5.3). The replication was pre-registered with fixed hypotheses, Holm correction, energy calibration rules, and non-inferiority tests, and the authors report results that go against them (the Gemma-2-2B replication fails; the pooled-audit accuracy gain does not replicate). This makes the reported numbers trustworthy even where the interpretation is debatable.

  2. A probe guarantee is shown not to be a behavioral guarantee (Tables 23–24). LEACE brings held-out probes to chance across fit sizes and shrinkage settings, yet leaves the patching effect unchanged, and its direction is nearly orthogonal to the contrast direction. This is a useful and practically relevant caution for work that evaluates erasure by probe accuracy alone.

Critical weaknesses
  1. The site effect is largely confounded with the amount of editing (Table 18, Tables 11 and 14). The span edit touches about 3.4 tokens, while the every-position edit touches about 220 tokens and removes roughly 50–100× more activation energy, so the damage from applying the direction everywhere is expected from the amount removed rather than from position as such. The random subspaces applied everywhere are not energy-matched either. The only energy-matched site comparison (H2, span vs. control span) passes on Llama-3.1-8B, where the authors themselves mark it as limited because the energies drifted beyond their 10% rule, and is null on Gemma-2-2B. This matters because the title and second contribution claim that the site, more than the direction or operator, decides the outcome.

  2. The scope is narrow and the main reduction is partly built into the instrument (Sections 3.1–3.3, Table 2, Appendix F). The claims are framed for linear concept erasure in general, but the evidence covers one attribute family that has an explicit token span, which the method requires. The direction is fitted from the same swap at the same positions that the patching instrument perturbs, so a lower patching effect after removing it there is largely expected. The independent readout, answer sensitivity T, improves significantly only on Llama-3.1-8B, and its per-seed changes are nearly uncorrelated with the patching-effect changes (r = 0.11 and 0.02).

Questions for authors
  1. Can you provide energy-matched site comparisons beyond the span vs. control span, e.g., the erasure applied at every position or at the last token with total removed energy scaled to the span edit, and random directions applied everywhere at the erasure's energy? Without these, how do you separate the effect of position from the effect of the amount removed?
  2. Was the "truncate to the shorter span" alignment rule fixed before any test results were seen? Table 27 shows that padding instead of truncation roughly doubles or triples the reduction on unequal-length pairs, so the choice materially affects the reported effect size.
  3. Is there any evidence that the site effect holds beyond demographic terms with an explicit token span? If not, would you narrow the title and claims to this setting?
  4. With 27 tables and two evaluations using different readouts and model sets, the main findings are hard to locate. Consolidating the key confirmatory results into one table in the main text would substantially improve readability.
Reviewer_JwYiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to apply activation edits that remove demographic information from language models. The experiments cover four models, with a separate replication on two models. The results show taht editing the demographic tokens reduces the measured dependence, but also removes useful information.

Critical strengths

The paper asks a useful question: where should an activation edit be applied, and what does it cost? The authors report negative results and accuracy losses, including cases where removing demographic information prevents the model from answering correctly.

Critical weaknesses

The conclusions are a bit too strong, noyt quite supported by the expriments. The behavioural benefit are on only one of the two models and depends on how answers are scored. It is hard to dsiti9nguish between reducing patching and reducing demographic influence on answers.

Detailed comments:

  • 073–085: What is new here? Is it the erasure method, the choice of where to edit, or the evaluation? The main tools already exist.

  • 129–155, 346–350: On Gemma, the patching effect decreases, but answer sensitivity does not decrease significantly. Thus, reducing the patching effect does not necessarily mean that the answers depend less on demographic information.

  • 180–196: You acknowledge that the labels used in the initial LEACE comparison do not describe a single concept. Well, then, what can we learn from LEACE not working in this setup?

  • 245–263: You select the demographic pair with the largest original patching effect for each seed. How much does this choice affect the results? Please also show results without this selection.

  • 279–287, Table 2: The reduction in answer sensitivity holds on Llama, but not on Gemma. Why?

  • 319–344: On Llama, the edits remove different amounts of energy, beyond your allowed tolerance. Can you repeat the comparison with matched amounts? Otherwise, we cannot tell whether the benefit comes from the edit position or the amount removed.

  • 361–374: Accuracy drops sharply when the demographic term is needed to answer the question. Weaker edits cause less harm, but also help less. Is there a setting that reduces dependence without losing useful information?

  • 379–391, Table 3: Additive steering works as well as, or better than, erasure, sometimes with better accuracy. What advantage does erasure offer?

  • 405–411, Table 25: LEACE trained with clear gender labels reduces the patching effect while removing much less energy. You need to compare the methods at similar strengths.

  • 455–459, Table 21: The Llama benefit disappears or reverses when you change how answers are scored. This raises doubts about your findings.

  • 468–479: The conclusion says that edit position decides the outcome. However, the results also depend on the model, scoring method, edit strength and training labels?

Questions for authors
  1. What is actually the main new contribution?

  2. Why does the benefit disappear under alternative answer scoring?

  3. Does the position effect hold when test-time energies are properly matched?

  4. Can you reduce dependence without removing needed information?

  5. What does erasure offer over additive steering?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.