- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues that in linear concept erasure, the token positions where a projection is applied largely decide what gets removed and what it costs. The authors measure a model’s dependence on a demographic term by patching activations between two versions of a bias item that differ only in that term. They then erase the resulting direction only at the term’s own tokens. Across four open models and a pre-registered replication on new BBQ items, this matched-position edit reduces the measured dependence, while swap-side LEACE and random subspaces do not, and applying the same edit at other positions is often ineffective or harmful. The authors also show a clear cost: the edit removes the demographic information even when the question needs it, which lowers accuracy.
Critical strengths
- The paper raises an interesting and practically relevant point: the position where an erasure is applied matters, which is something that existing erasure methods tend to leave implicit.
- The authors evaluate their method on several open models and benchmarks.
- The paper includes a fairly detailed appendix and releases code, which helps with reproducibility.
Critical weaknesses
I should mention that I am not an expert in this area, so please take the following points as my impressions and questions rather than firm conclusions.
-
I found the main effect a bit hard to interpret because it does not seem fully consistent. In the pooled audit, the reduction is not significant on Qwen2.5-7B, and if I read Table 9 correctly, only about half of the pairs show a lower dependence after the edit, so the average effect may come from a smaller group of items. In the pre-registered replication, the main hypothesis does not seem to hold on Gemma-2-2B, and on Llama-3.1-8B some of the tests are marked as limited by the authors. The Llama-3.1-8B result also seems to change with the scoring method (Table 21), and the accuracy gain on Gemma-2-2B in the pooled audit does not appear in the replication. I also noticed that the first token of the gold option often starts other options as well (Table 7), which might make the pooled audit’s measure a bit noisy. Could the authors comment on how robust they think the effect is?
-
I was a little unsure about the LEACE comparison. As far as I understand, the main LEACE baseline uses a label that mixes many different term pairs, which the authors themselves note is not one coherent concept, so its weak result might be partly expected. In Table 25, when LEACE or the mean difference is fitted on a coherent gender label, both seem to reduce the dependence, and on Gemma-2-2B the mean difference even appears to do slightly better than the proposed erasure. The erasure also seems to remove much more activation energy than LEACE in that table, which makes the “about twice as much” comparison a bit hard for me to read. It might help to compare against these stronger baselines at matched energy. Would the main conclusion about LEACE still hold in that setting?
-
Since the method needs to know where the demographic term appears in the prompt, I wondered whether a simpler approach, like replacing the term with a neutral word such as “person,” might work similarly. The results seem to suggest that the erasure behaves somewhat like hiding the term from the model, since accuracy drops a lot on items where the term is actually the answer (Table 15). A comparison with such prompt-level changes would help me understand the benefit of the activation-level edit. What do the authors see as the main advantage of their approach over simply rewriting the prompt?
-
The experiments use models with at most eight billion parameters, and the replication uses two of them. The study also focuses on one attribute family and needs the attribute to appear as explicit tokens, which the authors also mention as a limitation. I am curious whether the findings would carry over to larger or newer models, or to other types of concepts. Have the authors tried any such settings?
-
The presentation of the paper needs substantial improvement. The main text is very dense, relies on many custom terms (such as pooled audit, fresh replication, sequential source state, and calibrated cells), and reports almost all results through long tables packed with numbers. There is only one figure in the main text, so the key findings are very hard to grasp at a glance. The figures that do exist are also difficult to read: the text and legends are small and unclear, and in Figure 4 some data markers overlap with their text labels. Overall, the paper reads as if it was prepared in a hurry, and it would need a clearer structure, fewer custom terms, and well-designed figures summarizing the main results to be accessible to readers.
Questions for authors
See the weaknesses.