acceptodds
Withdrawn from ICLR 2027

Probe-Geometry Alignment: Auditing and Suppressing Representational Memorization in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unlearning methods for language models are judged by what the model says. Prompt the model with the opening words of a target passage, and if it no longer completes the passage, the content is reported as removed. A model can pass that test and still encode the passage. We measure what is left with a probe that has to generalize. A linear classifier is fitted on the hidden states of several memorized sequences, then tested on a memorized sequence it has never seen. To succeed, it must read a property of memorization itself rather than a particular string. It succeeds on every model we tested. Across three pretrained architectures, the memorization-specific gap runs from +0.19 to +0.35. It is stable across the Pythia family from 70M to 1B parameters, and it holds on a pool whose membership is verified by corpus lookup rather than by likelihood. The direction the probe reads is causally separable from the machinery that recites the text. No behavioral erasure we applied removes it, from five published unlearning objectives to weight editing to imitating a model that never memorized, however far the output moves. That separation is what makes an efficient method possible. Probe-geometry alignment (PGA) constrains one scalar per depth, the projection onto the direction the probe reads, and leaves the rest of the residual stream alone. On Pythia-70M it buys 6.6 times as much recall suppression per nat of capability as the best of four isotropic settings on the same recipe. It also suppresses more recall than the behavioral baseline that retains the most capability, while retaining more capability than that baseline, and it carries over to Mistral-7B. Constraining a readout can also reverse it rather than empty it. Raw probe accuracy then falls below chance while the classes stay separable, so an audit that quotes that number records erasure where none occurred. A multi-position variant drives a fresh mean-pooled linear probe to chance at mid depths, at a further cost in capability. The audit and the method share one direction, so a deployment able to fit a probe can state what its deletion claim is worth, and act on the answer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
memorization, machine unlearning, linear probing, interpretability, concept erasure, representation analysis, auditing, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_KWCeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is incomprehensible to read. There is no clear motivation, problem statement, or methodology. The paper is perhaps generated by AI.

Critical strengths

None

Critical weaknesses

I flag the paper as AI generated, which, in my opinion, is its critical weakness.

Questions for authors

What is the objective of writing the paper?

Reviewer_gRTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether language models still contain traces of memorized text after unlearning. The authors train a linear probe to distinguish memorized texts from control texts and test whether it generalizes to unseen sequences. They find that several unlearning methods reduce text recall but do not remove the probe signal. They then propose PGA, which updates the model to reduce this signal while preserving general capability, and evaluate it on Pythia, GPT-2, Mistral, and a small controlled transformer.

Critical strengths
  1. The paper studies an important problem. It clearly shows that behavioral measures alone can give an incomplete picture of unlearning, and that internal representations should also be examined.

2.The paper contains a lot of empirical work. The experiments cover several model families, model sizes, and unlearning methods.

Critical weaknesses
  1. The paper is hard to follow. The terminology keeps changing, many empirical results are presented before the relevant concepts are properly defined, and the overly defensive writing makes the main argument difficult to understand.

  2. The proposed method does not seem particularly novel. Prior work such as [1] Intrinsic Test of Unlearning Using Parametric Knowledge Traces(not cited) already argues that behavioral forgetting does not imply the removal of internal knowledge traces. The authors argue that PGA differs from existing concept-erasure methods because those methods assume a fixed, known concept, whereas memorization is neither fixed nor known and therefore requires the probe direction to be repeatedly refitted. This distinction is not convincing. In the paper’s setup, memorization is operationalized using fixed binary labels and only the best separating direction changes during training. Prior work such as INLP similarly refits a classifier after each direction is removed. Repeatedly refitting the probe therefore does not clearly distinguish PGA from prior concept-erasure methods.

  3. There is a clear logical leap in the motivation for PGA, together with a missing key ablation. Showing that later layers reconstruct a direction removed at one layer only establishes that single-layer projection is insufficient; it does not establish that model parameters must be updated. The paper should directly compare trained PGA with multi-layer inference-time projection and iterative methods such as INLP, R-LACE, or LEACE. Without this ablation, the main design choice behind PGA is not justified.

Questions for authors

See Weakness2 and 3.

Reviewer_SQJFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks the question of does suppressing a language model's ability to reproduce memorized text actually remove the information from its internal representations? The authors develop a cross-sequence leave-one-out probing protocol that trains linear classifiers on memorized sequences and matched controls, then evaluates their ability to generalize to held-out sequences. They find that a detectable representational signal often persists even when existing unlearning methods substantially suppress behavioral recall. They then propose Probe-Geometry Alignment (PGA) from these observations, which constrains differences along learned probe directions while preserving other components of the hidden representation. The authors report strong recall-suppression and capability-retention trade-offs relative to isotropic alignment, and show that orientation-corrected probe accuracy avoids misinterpreting inverted classifier predictions as erasure.

Critical strengths
  1. I think the paper addresses an important limitation of how machine unlearning is currently evaluated. A model failing to reproduce something does not necessarily mean it no longer contains that information, and looking at internal representations seems like a useful way to investigate this.
  2. The leave-one-out probing setup is interesting. Training on some sequences and testing on unseen ones is more convincing than simply showing that a classifier can distinguish one memorized passage from a control. I also appreciate that the authors include several controls to check whether the probe is picking up unrelated differences between the sequences.
Critical weaknesses
  1. I'm not fully convinced that the probe is detecting memorization specifically. Since the memorized sequences are largely identified using high likelihood, it seems possible that the probe is learning to recognize highly predictable text rather than memorization itself.

In Appendix F, even a randomly initialized Pythia-70M model has a positive memorization-specific gap of 0.188, compared to 0.323 for the pretrained model. I understand that the authors show different patterns across layers, which suggests that pretraining contributes something additional. However, it is still unclear how much of the measured signal can actually be attributed to memorization. The corpus-based experiment compares sequences with different levels of reproducibility rather than directly isolating the effect of memorization.

  1. The results show that PGA can change what the probes detect, but the information often remains accessible through other probes. For example, Table 2 reports a corrected probe accuracy of 1.000 after PGA at the deepest layer. Even with multi-position PGA, the strongest nonlinear attacker still achieves 0.789 accuracy at depth three, and the signal returns at deeper layers. The authors do acknowledge these limitations, but this raises the question of whether PGA is meaningfully reducing the stored information or mainly making it harder for a particular type of probe to detect.

  2. The reported 6.6 times efficiency improvement over isotropic alignment is strong, but most of this comparison is on Pythia-70M. Appendix H also reports a noticeably different efficiency ratio when changing the held-out evaluation set. It would help to have a comparison across multiple levels of recall suppression, rather than relying mainly on the ratio from particular configurations. It would also help to know how much the results vary across seeds and whether the advantage holds for stronger concept-erasure baselines. The authors mention INLP, R-LACE, and LEACE in Section 2 of the paper. It would be valuable to consider when evaluating PGA.

Questions for authors

Since the memorized sequences are primarily identified using high likelihood, how can we be sure that the probe is detecting memorization rather than just recognizing highly predictable text? More specifically, how do the authors explain the positive gap of 0.188 in the randomly initialized model? Maybe one way to do this is by comparing sequences with similar likelihoods but different levels of training exposure.

To what extent is PGA actually reducing the information stored in the model? Is there any evidence that the information becomes harder to recover more generally, rather than just harder for the particular probes used during alignment to detect?

How would PGA compare against existing concept-erasure methods such as INLP, R-LACE, or LEACE? The authors mention that these methods assume a fixed concept, but could they be adapted to update the erasure directions during training?

Reviewer_ocMQOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A

Critical strengths

N/A

Critical weaknesses

This paper is written with dense terminology akin to AI written text. I am unable to review it in its current form

  • “pure-distinguishability null,”
  • “cross-sequence memorization signature,”
  • “register-matched control,”
  • “orientation-corrected accuracy,”
  • “memorization-specific gap,”
  • “causally separable from the machinery that recites the text,”
  • “probe-geometry alignment,”
  • “the first-moment signal a readout uses,”
  • “separating the two axes,”
  • “the instrument and the method share one object,” and
  • “a constrained readout can turn over rather than empty.”
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.