acceptodds
Under review as a conference paper at ICLR 2027

Why Erased Embeddings Look Safe: Fitting the Eraser on the Rows You Release

Abstract

Concept erasure is usually evaluated by how far an attacker's accuracy on the erased representation sits above chance. We show that this number depends on two choices that papers rarely report: which family of attackers runs the probe, and whether the eraser was fitted on the same rows that are released. Owners typically fit the eraser on the data they hold, which is also the data they release. We prove that when the embedding has at least as many dimensions as released rows, this fit subtracts each row's own class centroid, and that for any number of rows it leaves a probe trained on one part of the release pointing against the rest. A sweep over seven embeddings, six ratios of rows to dimensions and three erasure methods confirms both consequences. Linear probes fall below chance at all 28 settings, which the standard metric reads as strong erasure. On ReLU embeddings, where more than half of the coordinates are exactly zero, the same fit turns every zero into a class-specific constant, and a depth-4 tree recovers the protected attribute at 0.79 above chance while linear probes sit 0.20 below it. Fitting the eraser on disjoint rows removes both effects. The mechanism also explains why, on 109 evaluation cells with three published erasure methods at matched utility, two attacker families of equal size disagree about whether to release on 9.1% of cases. The disagreement recurs on 73 new cells, and it extends to method rankings: a kernel-based eraser leaks less than LEACE under one attacker family and no less under the other.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
concept erasure, embedding privacy, attribute inference, evaluation methodology, adversarial evaluation, LEACE, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Aq7EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether erased embeddings are actually safe. It shows that evaluation by existing methods depends on the probe and how it is fitted. The authors run a sweep over multiple encoders, datasets, and three erasure methods with two attack families: family R (logistics, MLP) and family T (gradient-boosted trees, k-NN). Their results show significant decision disagreement across attack families. The main point is if the owner release the same rows of embeddings that are used to fit the eraser, the embeddings may quietly handle the private attribute to a different probe (e.g, a tree). The paper recommends data splits (fitting the eraser on disjoint rows) so that it is easier to interpret erasure results.

Critical strengths

A shallow tree catching what the linear probe misses is interesting (Tied coordinates finding about linear erasure handling the attribute to a shallow tree). The linear probe accuracy can be easily read as safe due to below-chance accuracy, but it is due to the same fitted rows. It reinterprets below-chance accuracy. The fix is cheap and actionable.

The experimental controls are careful such as equal-size attacker, floor correction measures, a designed table, replication on new cells, a matched-n control, etc. And it tested a published conclusion directly.

Critical weaknesses

Contribution 1 is mixed with the protocol and the phenomenon. A similar work by (Snoek et al. 2019, https://www.sciencedirect.com/science/article/pii/S1053811918319463) in neuroimaging data shows that confound regression produces a negative bias below-chance in realistic cases, the bias disappears when the confound regression is done inside cross-validation fold. It is structurally similar to fitting on disjoint rows. The novelty will strengthen if the paper explains what is new in concept erasure settings. There is also a similar recent work in vision-language models about nonlinear bias survival (https://openreview.net/pdf?id=x45Ho75YmJ). The core measurement overlaps, but the settings are different.

The presentation is difficult to read. There are a lot of results. The highlights are too short. They are not easily understandable. The main content is not standalone. For example, in section 4, the ten statements (H1-H10) are from the appendix, but they are referenced throughout the section. Appendix should be additional material not the essential parts. In appendix, there are a lot of code scripts references, but there is no code attached in the supplementary material or anonymous repo.

Questions for authors

Does disjoint fitting remove the disagreements in all cells, or only most?

1 comment

Thank you for the Snoek et al. reference, which we had missed; it is now cited and the novelty statement is rewritten around it.

Snoek et al. (2019) and what is new here. The structural match is real: regressing a nuisance variable out of all rows before cross-validation biases accuracy downward, below chance in the limit, and fitting inside each training fold removes the bias. Our identity (ii) is the case of that analysis in which the variable regressed out is the attribute itself, and disjoint fitting is the per-fold fix; §2 now says this in so many words, with Jamalabadi et al. (2016) for below-chance small-sample classifiers. What those accounts do not contain: the exact form of the fit at n≤d+1n \le d+1 (in-sample LEACE subtracts each row's own class centroid); the opposite-sign effect, by which the same fit makes the attribute more readable to a depth-4 tree through tied coordinates (0.786 above majority on ResNet-50 FairFace against 0.104 under disjoint fitting); the consequences for release decisions and method rankings; and the point that in the release setting the in-sample fit is not an analyst's slip but the deployment, so the per-fold fix changes what is released. Singh & Hegde (2026) is now cited as concurrent work: it measures the stronger-probe fact on vision-language embeddings and does not involve the fitting protocol.

Code. You are right that nothing was attached: the archive (code, dated analysis specifications, recorded results, and the scripts that produce every number, table and figure) was only linked, although the statement called it supplementary material. It is now attached to the revision as supplementary material, including the scripts and results of the new disjoint-fit arm.

All cells or most? In the controlled sweep, all: with each family reduced to its strongest member, 10 of 28 settings disagree under in-sample fitting and 0 of 28 under disjoint fitting (H9). On the designed table we have now run the arm you ask about, pre-registered before the run (revised PDF, §4 "Disjoint fitting on the flipped cells" and Appendix X10): the 19 cells that carry the 22 flips, same seeds and released rows, eraser fitted either on the released rows or on other rows of the same cache. In-sample, 19 of the 22 flips reproduce. Under disjoint fitting 2 of those 19 disagreements remain (both CelebA CLIP, and in the opposite direction: the linear family refuses what the tree family releases); the disagreement rate falls by 0.49 [0.25, 0.74]. So on the designed table the answer is "most, and the remainder is not the in-sample signature". Two honest limits. First, on the 8 ResNet-50 pairs the families agree under disjoint fitting but both refuse (R 0.06–0.16, T 0.07–0.18 above floor); we had pre-registered that T would fall below the threshold and it did not, so the erased ResNet-50 embeddings leak under both families rather than being safe. Second, the arm rests mainly on LEACE: 10 SPLINCE and 14 LEOPARD cell–arms do not reach LEACE's task cost under disjoint fitting on enough seeds and are excluded by the pre-registered rule.

H1–H10. Now tabulated in the main text (Table 1) and named where used; the gate study's detail is in an appendix.

Reviewer_bEpqOpenReview ↗
Rating
1(−1)
Confidence
2(−2)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper raises an important question: the measurement of concept erasure can heavily depend on the evaluation protocol, especially the choice of attacker family and whether the eraser is fitted on the same rows that are later released. In particular, the paper shows that in-sample eraser fitting, although seemingly innocent, can introduce algebraic artifacts into the evaluation. One valuable theoretical observation is that, after in-sample erasure, an attacker trained on one split of the released rows can systematically learn a direction that is harmful on the test split, which explains why linear probe accuracy can fall below chance. The paper further shows that tied coordinates, which are common in sparse ReLU embeddings, can become class-specific constants after erasure and therefore leak the protected attribute to nonlinear attackers. The paper provides extensive empirical evidence across different embeddings, erasers, attacker families, and fitting protocols to support these observations.

Critical strengths
  1. The core phenomenon is interesting and practically relevant: fitting an eraser on the same rows that will later be released can create pathological evaluation behavior and substantially affect measured leakage.

  2. The paper provides a valuable explanation for the below-chance linear probe accuracy under in-sample eraser fitting. In particular, an attacker trained on one split can learn a direction that becomes systematically harmful on the complementary test split.

  3. The theoretical analysis is solid and insightful. The two algebraic identities provide a clear explanation for important empirical observations rather than leaving them as unexplained experimental artifacts.

  4. The tied-coordinate phenomenon is particularly interesting. The paper shows that tied coordinates, which are common in sparse ReLU embeddings, can become class-specific constants after erasure and leak the protected attribute to nonlinear attackers. The quantization experiments further support that ties, rather than ReLU itself, are responsible for this effect.

  5. The empirical evaluation is broad and transparent. The paper studies multiple embeddings, erasers, attacker families, fitting protocols, and sample-to-dimension ratios, and it also reports failed hypotheses and weaker replication results rather than only presenting positive findings.

Critical weaknesses
  1. The paper’s readability and presentation needs substantial improvement. From the abstract, it is hard to follow the concepts, and the reader, including me, may have questions such as “What is the concept ‘erasure’?”, “What is the definition of a probe?”, and “What are ‘rows’ in this context?”. The organization of the paper is also hard to understand. The paper is dominated by designed evaluations and experimental protocols/details, which makes the main claim difficult to identify. The paper needs substantial rewriting.

  2. The paper is also insufficiently focused. It is unclear whether the main claim is that “attacker family changes concept-erasure evaluation”, that “in-sample fitting produces algebraic artifacts and tied-coordinate leakage”, or a broader measurement-oriented study. Among these, the in-sample fitting phenomenon and its theoretical explanation appear to be the strongest contribution, but this is not sufficiently emphasized in the current organization.

  3. Section 5 is particularly distracting and needs substantial rewriting. The large number of gates, rungs, criteria, and experimental details makes the section difficult to follow, while its connection to the central contribution is not sufficiently clear. Overall, the scientific communication of the paper is insufficient.

Questions for authors

How common is the in-sample fitting protocol in existing concept-erasure evaluations or realistic deployment settings? More broadly, to what extent do the main observations here generalize beyond the studied erasers and concept-erasure settings within this paper’s scope?

1 comment

Thank you for identifying the in-sample mechanism as the contribution; the revision is reorganised around it (top-level comment, points 1–5): the mechanism is now §3 and opens with the threat model, the designed table follows as its consequence, and the gate study is reduced to one paragraph with the detail in an appendix.

How common is in-sample fitting? In method papers, evaluation is disjoint: LEACE's own experiments fit and evaluate on separate slices, and Naowarat et al. (2026) note that prior work often fits subspaces and probes on one set. In deployment the question answers itself: the owner releases the rows it holds, which are the rows it fitted on, so the released artefact is the in-sample case. That mismatch between how erasers are evaluated and what is shipped is the paper's point (§3, "The scenario"; recommendation 1). Neuroimaging has the same practice under another name, whole-dataset confound regression, with the same below-chance consequence (Snoek et al., 2019), now cited in §2.

Generalisation beyond the studied erasers. Identity (ii) needs only that the eraser equalise class means on the fitted rows, which every linearly guarding eraser does, so the below-chance reading applies to INLP, RLACE and SPLINCE as well; the sweep confirms SPLINCE. Identity (i) needs a whitening projection at n≤d+1n \le d+1: LEACE and SPLINCE show the tied-coordinate leak, SAL (unwhitened) does not. Whether INLP, RLACE, KRaM, LEOPARD or Obliviator show the tie leak is untested and §6 says so. Obliviator, fitted in-sample on the designed table, barely moves any release decision, which is consistent with the identities not covering it.

Reviewer_mERTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper propose several protocols for evaluation to test the effectiveness of erased embeddings. The paper considers popular linear erasure techniques and evaluates them using different linear & non-linear classifiers. Through experimentation, the paper observes several patterns of evaluation gaps using existing erasure techniques and proposes a set of proposals to rectify them.

Critical strengths

N/A

Critical weaknesses
  1. The paper in its current form is very hard to understand. The paper uses terminologies not consistent in the literature, abruptly introduces terms or phrases without describing them. The protocols presented at the end are the result of testing a set of hypotheses that not even presented in the main paper. These are written in the appendix and requires a lot of human effort to go-and-back and understand them. Please find a few examples of the phrasing that makes it hard to understand in the comments below.

  2. The premise of the contribution is also unclear. Linear erasure techniques work under the assumption that the attack is done by a linear adversary (prediction network). The authors find out shortcomings by evaluating these embeddings using non-linear classifiers like gradient boosted trees or k-NNs.

  3. Although cited in the paper, the paper skips comparison with strong non-linear erasure techniques like KRaM. It would be interesting to see if the current findings hold for these methods. Apart from this, previous papers have already talked about the fundamental limitations of concept erasure techniques (https://arxiv.org/abs/2503.20098). The paper should position their work w.r.t. to this work as well.

  4. Overall, the paper in it's current state requires significant work to improve its presentation and make it more comprehensible to the readers. The paper should also include non-linear erasure baselines and position their work appropriately w.r.t. existing works.

  5. Comments where the text is unreadable below:

Line 36: What does the phrase “SPLINCE keeps the linear removal” mean?

Line 46: What “cell” is being referred to over here?

Line 57: What does “shallow tree reads off” mean?

Line 59: What “rows” are being referred to here?

Line 75: What does “floor-corrected attacker families” mean?

Section 3: Heading, what does “Instrument” mean? This term hasn’t been defined or used so far.

Line 142: What does “panels” mean? Is it referring to the models?

Line 143: Which text encoders are used?

Line 144-145: The definition of “strata of affinity” is not clear.

Line 146: What does this mean? “how far the task label is determined by the sensitive one”

Figure 1a is not human readable at all. The origin of the plot is not at (0,0). The family R and T defined in text had several classifiers. Which one does each point belong to?

Line 245: what does “exact algebraic signature” mean?

Line 247: Why is caching important here? What is n and d? What are fitting protocols? I’m unable to understand the statements or their intuitions presented in H1-H10.

Section 4 talks about the results of different hypothesis statements without mentioning them. Therefore, it is not possible to follow the results in the main paper.

Line 294-95: What does sparse cache mean?

Line 307-308: What does “attacker reading” mean?

Line 364: Which “designed table” is the text referring to?

Line 372: R was used to describe a family, now it’s being refer as spearman correlation. Please use different notations. What does “rung” mean?

Questions for authors

Please respond to the questions in the weakness section.

1 comment

Thank you for the detailed list of unclear passages; every item in it is addressed in the revised PDF (points 1–6 of the top-level comment), and we answer the two substantive points here.

Premise (W2). The paper is not a demonstration that nonlinear probes beat linear erasure, which §1 already states is known and claims no novelty for. The central result is a failure of the linear reading itself: when the eraser is fitted on the rows it releases, the linear probe, the adversary the guarantee is about, reads below chance at all 28 settings of the sweep (H7, 7 of 7 caches), and an accuracy-above-majority metric counts that as safety. The two identities in §3 say exactly why. Two of the four erasers we run target nonlinear readout (LEOPARD, Obliviator), and the ranking reversal in §4 is between LEACE and Obliviator (12 of 32 resolved cells). The threat model is now the first paragraph of §3: the owner fits on the rows it holds and releases those rows; the recipient trains whatever model it likes.

KRaM and positioning (W3). KRaM is cited and not run; LEOPARD (MMD) and Obliviator (kernel) are the nonlinear erasers in the table, and we now list KRaM with INLP and RLACE as untested in §6. Nothing we claim depends on which nonlinear eraser is used: identity (ii) needs only that the eraser equalises class means on the fitted rows, which every linearly guarding eraser does; identity (i) needs a whitening projection. Chowdhury et al. (2025) bound what perfect erasure can achieve in population at a given utility; our concern is the finite-sample estimate an owner reads on the rows it releases. The two are complementary and the revision says so in §2.

Definitions you asked for. nn is the number of released rows and dd their dimension; the identities are conditions on n/dn/d. A cache is a fixed matrix of frozen-encoder embeddings, so that across protocols only the rows the eraser is fitted on change. The three protocols are in-sample (fit on the released rows), disjoint (fit on other rows of the same cache) and two-fold cross-fit. A family's reading is its best member's test accuracy minus the same best-of-family accuracy on permuted labels, so the best-of-six selection is matched rather than counted as leakage. "Rung" is a level of an attacker tuning ladder fixed in advance. Figure 1a has one point per cell–method pair, located at the two families' readings; the axes extend below zero because in-sample fitting depresses family R's reading (§3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.