acceptodds
Under review as a conference paper at ICLR 2027

Does Removing a Prediction Remove Its Encoding in Language Models?

Abstract

To erase harmful or protected content, unlearning often suppresses a prediction and assumes the encoding behind it disappears together. A surviving encoding keeps that content recoverable. Yet it is not well understood whether this encoding, the internal distinction between tokens, needs its own prediction to form or persist. We measure it as identity dispersion, the spread among mean representations before each token of a category, such as a part of speech. We test this causally in four models up to 1.5B parameters, removing the prediction in four ways, from scratch or after pretraining, against identical unconstrained twins. Removal provably raises prediction loss by at least what the prediction gained from context, so the prediction is gone. In every removal, at least half the categories keep 94% or more of the twin's dispersion one layer below the output, and with one exception no category's median falls below half. Trained without the prediction, models stay 58 to 73% as decodable by a linear probe as their twins. Across categories, the fraction of variance separating a category's tokens tracks the information its prediction uses, and a law we fit on four models predicts it in six other model families. Removing a prediction does not necessarily remove its encoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, representation geometry, causal interventions, information theory, neural collapse

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ELw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether suppressing token distinctions at the output-logit level also removes those distinctions from internal representations. Through experiments conducted both from initialization and after pretraining, with matched unconstrained controls, the authors examine whether these distinctions form or persist, what might sustain them, and whether the remaining information can be decoded. They find that substantial internal distinctions remain despite the output constraints.

Critical strengths
  1. Clean intervention. Sharing output weights within each category enforces context-independent within-category probability ratios, ruling out incomplete suppression of the targeted prediction channel.

  2. Extensive empirical validation. The paper examines multiple intervention settings and uses complementary measurements, including whitened dispersion and linear probes, to assess whether internal distinctions remain and are decodable.

Critical weaknesses
  1. The presentation makes the paper difficult to follow and its contribution difficult to assess. 1.a. Too much terminology is introduced without a clear explanatory sequence. Definitions and operational details are scattered across sections, requiring readers to reconstruct the setup. For example, the second paragraph of the introduction introduces identity dispersion, readout channels, formation, maintenance, and unconstrained twins in rapid succession, before clearly explaining the concrete intervention. 1.b. The experimental narrative and presentation of results need clearer organization. The experimental descriptions are often difficult to parse, and experiments are presented in succession without sufficiently explaining their connections. Figure 2 and Table 1 have dense captions, while essential explanations are deferred to later sections or the appendix. Readers must repeatedly move between sections to understand the interventions, measurements, and conclusions. Simpler descriptions, concrete examples, and essential methodological details in the main text would make the evidence easier to assess.

  2. The conceptual contribution and its connection to unlearning are insufficiently established. 2.a. The intervention and measurement concern different distinctions. The intervention removes output-token distinctions but leaves systematic semantic differences between their preceding contexts intact. Identity dispersion can therefore capture ordinary contextual differences, such as sunlight versus moisture, even when the corresponding output distinctions are suppressed. The paper does not sufficiently explain what the persistence of this measure reveals beyond continued contextual representation. 2.b. The motivating connection to unlearning remains underdeveloped. The experiments neither evaluate an unlearning method nor test the recovery of targeted knowledge. The paper needs to explain how retained contextual distinctions inform the motivating problem of whether forgotten content remains recoverable. Without this connection, the broader significance of the findings remains unclear.

Questions for authors

See the Weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ELw4 (1 of 2)

Thank you for the careful reading. We are glad you note the clean intervention and the extensive empirical validation.

1.a "the second paragraph of the introduction introduces identity dispersion, readout channels, formation, maintenance, and unconstrained twins in rapid succession, before clearly explaining the concrete intervention."

The introduction (page 1) now explains the intervention before it names any of the five terms you list. It walks through one example (rainy against sunny) and only then names the prediction channel (formerly the readout channel), maintenance, formation and the twin, an identical run without the removal. Identity dispersion follows in the next paragraph. Section 2 ends with a table of the core terms (page 4). We audited the main text for technical terms used before their definition, and replaced terms with plain language where they were not needed. For example, the submission used "treated", "usable" and "whitened dispersion" before defining them. Each is now defined at or before first use. We also unified terms that refer to the same concept: in the main text, the timing has one name (from initialization), and a run under an intervention is always a treated run.

1.b "experiments are presented in succession without sufficiently explaining their connections. Figure 2 and Table 1 have dense captions ... Simpler descriptions, concrete examples, and essential methodological details in the main text would make the evidence easier to assess."

We made the three changes you suggest, and Section 4 now connects the experiments as four questions, each raised by the answer before it (Sections 4.1 to 4.4). Does the encoding persist after pretraining? Does it also form without features learned before the removal? Is what survives usable? What sustains it?

  • Simpler descriptions. Table 1 (page 6) now reports two measures, whitened dispersion (the spread between token means relative to that within each token) and the linear probe, so its caption explains two instead of six. Its other ratios moved to Table 14 in the appendix. Figure 2's caption (page 3) now says what each marker in panel (a) shows.
  • Concrete examples. Beyond the weather example on page 1, step 2 of the procedure shows the removal on months: the model can still predict that a month comes next, but not whether it is March or May (page 5). Section 4.3 adds a test on 60 month facts (new, exploratory).
  • Essential methodological details in the main text. Section 4 now gathers the whole procedure in six numbered steps (page 5): categories, removal, comparison with the twin, timing, the four interventions and what each removes, and the measures.
AuthorsOpenReview ↗

Response to Reviewer ELw4 (2 of 2)

2.a "Identity dispersion can therefore capture ordinary contextual differences ... The paper does not sufficiently explain what the persistence of this measure reveals beyond continued contextual representation."

Persistence shows more than a network representing its input: it marks a learned distinction that other predictions use and from which the removed choice among a category's tokens can be partly read back (Section 4.4, "Why persistence is informative"). Two results from the submission show this, and the revision adds one baseline:

  • In a controlled task, context alone does not sustain the distinction: it survives only when another prediction reads the same attribute (in the submission). In a synthetic corpus, the treated category's next token depends on an attribute of the previous token, and two versions differ only in whether a partner category reads the same attribute or an independent one. With its own channel removed, the treated category keeps 0.47–0.53 of its twin's identity dispersion in the first version and 0.001 in the second (Figure 2b shows the same contrast on a log scale).
  • In the language models, other predictions use the distinction, and the removed choice can be decoded from it (in the submission; Sections 4.3 and 4.4). In models trained without the prediction, a linear probe for the removed choice recovers 58–73% of what it recovers in the twin, counted above an untrained network. Erasing the directions that separate a treated category's tokens raises other predictions' loss 3.0 times as much as erasing random directions of the same size in the median cell, and 1.3 times when the change to the states is matched.
  • The distinction is learned (baseline added in the revision). In an untrained network fed the same contexts, the fraction of a category's variance that separates its tokens is 0.02–0.05, against 0.09–0.12 after training.

For interpretability, a distinction in a model's representations therefore need not serve its own prediction, because another prediction can sustain it (Section 7).

2.b "The experiments neither evaluate an unlearning method nor test the recovery of targeted knowledge. The paper needs to explain how retained contextual distinctions inform the motivating problem of whether forgotten content remains recoverable."

The recovery test you ask for, now in the revision, answers the motivating problem: targeted knowledge stays partly recoverable after its prediction is removed (new, exploratory; Section 4.3, Appendix D.21). In every model one of the 50 categories holds the twelve month names. We remove its prediction channel from pretrained models and score 60 month facts ("The Titanic sank on the 15th of"), restricted to that category's tokens. No prompt contains its answer, so the answer the hidden states give cannot be read off the input text.

In the two models whose twins answer most of the facts, the output gives one constant month, whereas the hidden states still answer more than half of the facts the twin gets right. We read the hidden states through the twin's month weights, with no fitting. In the other two models the twin itself answers few of the facts (37% and 18%; Table 13).

In Qwen2.5-1.5B the facts are partly recoverable even without the twin's weights. A probe fitted on generic text that contains none of the prompts gets 34% from the treated model, about half of the 69% it gets on the twin.

Unlearning should therefore be verified in the representation as well as at the output, since suppressing an output does not show that the content is gone. We test the clean case of output suppression by design: the choice is gone by construction, so the surviving trace cannot be a partly removed prediction, an explanation a partial removal leaves open (Section 6). Section 7 draws this consequence, and the abstract now ends on it.

Month facts in those two models:

Model (half the categories treated, two seeds)Twin, all 60 factsTreated model's own output (one constant answer; chance 6–8%)Its hidden states, twin's month weights, no fitting: all 60 factsOf the twin's correct facts, answered from those hidden states
Qwen2.5-1.5B84%10%62%68%
Pythia-1.4B55%10%32%55%

Dear Reviewer ELw4,

Thank you for prompting a direct test of the recovery of targeted knowledge. We are glad to report the result: facts the model no longer outputs remain partly readable from its hidden states (Section 4.3, Table 13).

The discussion period is shorter than last year's and ends soon, on November 18. We would be glad if this and our earlier replies have resolved your concerns. Should anything remain open, we would welcome the chance to address it in the time left.

Reviewing is unpaid work on top of your own research, and we thank you for the thought you have given our paper.

Best regards, The authors

Reviewer_XPoVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper examines how the variance among mean contextual representations preceding different tokens within a category changes when the model can no longer use context to choose which token to predict within that category

Critical strengths
  • The research question is interesting and has implications for understanding language model learning dynamics/representation geometry and for development of machine unlearning techniques
  • The causal intervention analysis gives an intuitive explanation of when/why this dispersion remains
Critical weaknesses

I had great difficulty understanding the paper due to its presentation. First, it introduces a lot of new jargon, which makes it difficult to follow. But more prominently much of the procedure is spoken about informally. So I just often couldn't tell what was being done. This critique applies to most of section 4. Because of this, I can't confidently support or defend many of the experimental choices. The listed strengths are coming from the authors descriptions of results, rather than my own verifications

Questions for authors

I was missing more of a connection to/implications for unlearning, especially since that's what the abstract starts out with

5 comments
AuthorsOpenReview ↗

Response to Reviewer XPoV (1 of 2)

Thank you for your constructive comments, which shaped our revision, especially of Section 4. We are encouraged that you find the research question interesting, with implications for learning dynamics, representation geometry and machine unlearning.

W (jargon). "First, it introduces a lot of new jargon, which makes it difficult to follow."

We audited the main text for technical terms used before their definition, and replaced terms with plain language where they were not needed. For example, Section 7's "A circuit writing a distinction into the residual stream" now reads "a distinction written into a model's representations". We also unified terms that refer to the same concept. The submission called one timing "from scratch", "from the start of training" and "from initialization", and the revised main text calls it "from initialization", defined as the first training step. The introduction now explains the intervention on a weather example, with Figure 1 (page 2) drawing it in one picture, before it names any term (page 1), and Section 2 ends with a table of the core terms (page 4). Terms the submission used before defining them are now defined at or before first use, for example:

  • Treated run: a run under one of the interventions.
  • Usable category: a category with at least 200 validation occurrences.
  • Whitened dispersion: the spread between token means relative to that within each token.

W (procedure). "But more prominently much of the procedure is spoken about informally. So I just often couldn't tell what was being done. This critique applies to most of section 4. Because of this, I can't confidently support or defend many of the experimental choices."

Section 4 now specifies the procedure exactly, which consists of six numbered steps ("Design", page 5):

  1. Categories. kk-means groups token types into 50 categories by the pretrained model's average state before each type.
  2. Removal. In a treated category, every token's score becomes one shared weight vector applied to the state, plus a trained per-token offset. The shared term cancels in the choice among the category's tokens (the softmax's shift invariance), so that choice no longer depends on context. The model can still predict that a month comes next, but not whether it is March or May.
  3. Comparison. Each treated run has a twin trained identically except for the removal. Dispersion results are medians over categories of the treated-to-twin ratio, and a median of 0.5 or less (halving, set in advance) would mean the encoding needs its own prediction.
  4. Timing. Removal after pretraining (two epochs) tests persistence, and removal from initialization (three epochs) tests formation.
  5. Interventions (the paper's "arms"). Each removes something the encoding might need. Treating half the usable categories removes each one's own prediction, and each is a control in the other run. Treating all of them also removes the other usable categories' choices. Withholding the loss at their positions removes the training signal there. Treating the whole vocabulary makes every within-group choice context-free.
  6. Measures. One layer below the output, identity dispersion (how far apart the average states before a category's tokens are) and its whitened form. At the output layer's input, the held-out loss reduction of a linear probe that predicts which of the category's tokens comes next.
AuthorsOpenReview ↗

Response to Reviewer XPoV (2 of 2)

W (verification). "The listed strengths are coming from the authors descriptions of results, rather than my own verifications"

The revision makes the results easier to track without relying on our descriptions. Three ways to check them:

  • One number, end to end. Table 1's first entry, 0.86 (half the categories treated, after pretraining), means the typical category keeps 86% of its twin's whitened dispersion, far above halving. It is the median of step 3's treated-to-twin ratios from 18 runs (Appendix D.5): three models, two seeds, and per seed two treated runs on complementary halves plus one twin.
  • A check built into the experiment. The removal costs almost exactly what Section 3 predicts, the loss the prediction had gained from context. Over 246 cases (a treated category in one model and seed), the measured rise tracks that amount almost one for one (slope 1.047, correlation 0.962; Figure 3), while untreated controls stay near zero.
  • Code and records. In the supplementary material, PROVENANCE.md maps each result to the script that produced it and the record it reads.

Q. "I was missing more of a connection to/implications for unlearning, especially since that's what the abstract starts out with"

Our intervention tests an assumption often made in unlearning, that suppressing a prediction also removes the encoding behind it, and the assumption fails here. It is the clean case of that suppression, since by construction the model's choice among a group of tokens can no longer depend on context. Two results show the failure, one reported in the submission and one added in the revision:

  • The removed choice stays decodable (in the submission; Section 4.3). In models trained this way from initialization, a linear probe still recovers 58–73% of what it recovers from the twin, so more than half of what the probe can read about the removed choice survives.
  • Specific facts stay partly readable (added in the revision; exploratory; Section 4.3, Table 13). We test 60 month facts ("The Titanic sank on the 15th of") after removing the month choice from pretrained models, with half the categories treated. In the two models whose twins answer most of the facts, the output gives one constant month, whereas the hidden states still answer more than half of the facts the twin gets right (Table 13 gives every model and arm). We read the hidden states through the twin's month weights, with no fitting.

Unlearning should therefore be verified in the representation as well as at the output (Section 7; the abstract now ends on it), since suppressing an output does not show that the content is gone.

For those two models, the table gives the twin's accuracy, the treated model's constant answer, and what its hidden states answer, over all 60 facts and over those the twin gets right.

ModelTwin, all 60 factsTreated model's own output (one constant answer; chance 6–8%)Its hidden states, twin's month weights, no fitting: all 60 factsOf the twin's correct facts, answered from those hidden states
Qwen2.5-1.5B84%10%62%68%
Pythia-1.4B55%10%32%55%

Dear Reviewer XPoV,

Thank you for telling us where the paper was hard to follow. Section 4 now opens with the whole procedure in six numbered steps (page 5), and a table gathers the core terms (page 4). On your question about the connection to unlearning, a new test shows that facts the model no longer outputs remain partly readable from its hidden states (Section 4.3).

This year's discussion is shorter than last year's, and it ends soon, on November 18. We hope these changes address your concerns. We look forward to your reply and would be happy to answer any further question.

We realize this review came out of time volunteered from your own work. Thank you for the time and effort you put into evaluating our manuscript.

Best regards, The authors

Reviewer_4YZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether a model still distinguishes tokens internally once it can no longer use context to predict between them. The authors tie the output weights of tokens in a category and compare against an identical untied model. The distinction between members of the same categories survives, and a probe recovers 58–73% of it. So blocking a prediction does not remove it from a model's latent representation.

Critical strengths
  1. Clean, complete removal of the prediction. Tying the output weights guarantees the within-category choice cannot use context. Unlike prior audits of unlearning and model editing, surviving information cannot be explained by leftover prediction. This matters because it is what makes the main finding credible, and it is the paper's main advance over prior work.

  2. The surviving information is decodable. In models trained from scratch without the prediction, a linear probe still recovers 58-73% of what it recovers from identical untied twins. This matters because it gives the result practical weight for unlearning: blocking an output does not make the information unrecoverable.

Critical weaknesses
  1. The headline result overstates retention. The abstract leads with a Euclidean centroid-spread figure (median category keeps >=94%), but centroid distance is not decodability. Centroids can be arbitrarily close yet perfectly separable (low noise along the separating direction), or far apart yet poorly separable (high noise along it). The paper's own scale-invariant measures, which do reflect decodability, show 56-86% (whitened dispersion, Table 1) and 58-73% (probe). This matters because the main claim, as stated in the abstract and Figure 2a, rests on a measure that doesn't test it. The conclusion survives, but it is weaker than presented.

  2. The paper is very hard to follow, and it's unclear what was tested. Key claims are packed into dense sentences with many new terms. The experimental design is a simple 2x2 (half vs all categories tied; from scratch vs fine-tuned) plus many additional conditions, but it is described through jargon labels. More importantly, the paper defines three very different category groupings (k-means at K = 10, 20, 50; POS; WordNet supersenses). They differ in what they capture and in the number of categories, and a token can belong to several. Yet the main text does not say which grouping the tying experiments use, and Section 5 pools results across groupings. This matters because readers cannot tell what was tested or whether the findings hold for each grouping, which limits both the paper's impact and how far its claims can be checked.

Questions for authors
  1. Which category grouping defines the tied categories in each Section 4 experiment? Since a token can be tied in only one grouping per run, please state this per condition. Where results are pooled across groupings (e.g. Section 5, "pooled over every partition"), please justify it. The groupings differ a lot: they capture different things (geometry for k-means, grammar for POS, meaning for WordNet), they have different numbers of categories (up to 80 for k-means across three K values, 11 usable for POS, and a different number again for WordNet), category sizes differ (e.g. all nouns vs a single K=50 cluster), and the same token can appear in several. A pooled median is then dominated by whichever grouping has the most categories. Reporting results per grouping would show whether the findings hold for each.

  2. Given that centroid distance does not measure decodability, why is Euclidean dispersion the primary measure rather than whitened dispersion or probe recovery? Would you revise the abstract and Figure 2a to lead with the invariant measures?

  3. Does tying change category-level prediction p(S|h), which it restricts to a single direction u? Does the surviving dispersion remain if u is projected out of h? If it doesn't, the distinction may sit in the category-level prediction rather than surviving independently.

  4. Prior work finds that facts removed by model editing (Patil et al., 2024) or unlearning (Hong et al., 2025) remain in hidden states, and the paper attributes this to possibly incomplete removal. Your experiments test broad token categories (e.g. parts of speech or clusters) rather than specific facts or concepts. Do you expect the result to hold at that finer level? If not, does complete removal still explain the prior findings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4YZT (1 of 3)

Thank you for a rigorous review, whose questions led to new analyses now in the paper (Q1, Q3, Q4). We are glad you find that the complete removal makes the main finding credible and is the paper's main advance over prior work, and that the result has practical weight for unlearning.

W1 and Q2. "The headline result overstates retention ... centroid distance is not decodability ... Would you revise the abstract and Figure 2a to lead with the invariant measures?"

The results hold for whitened dispersion, and the abstract and Figure 2(a) now lead with it as you suggest: the abstract's Euclidean 94% is replaced by the whitened 56–86%. Euclidean dispersion had led because Section 3's bounds are stated for it. On whitened dispersion (the spread between token means relative to that within each token; Table 1), a condition holds if the median category keeps more than half of its value in the twin, an identical run without the removal:

  • Persistence after pretraining (fine-tuned) holds: 85–86% in your 2×2, 81% in Qwen2.5-1.5B and 68% after six epochs.
  • Formation from initialization (from scratch) holds: 73–80% in your 2×2.
  • Loss withheld and whole vocabulary hold: 56–76%.

On your point that centroid distance is not decodability, we measured decodability directly: it agrees with whitened dispersion, and neither falls to half, so the finding does not depend on the choice of measure or of coordinates. In models trained from initialization, a linear probe for the removed choice recovers 58–73% of what it recovers from the twin, within 0.117 of the whitened ratio in every arm (bound set in advance at 0.12; Section 4.3). After pretraining, the probe reads 92% of what it reads from the twin (new, exploratory).

W2 (presentation). "Key claims are packed into dense sentences with many new terms. The experimental design is a simple 2x2 (half vs all categories tied; from scratch vs fine-tuned) plus many additional conditions, but it is described through jargon labels."

Our approach was to state the design in your terms before any label appears, and to give every concept one name, defined before it is used. The revision carries this out in four places:

  • Table 1 (page 6) now opens with your 2×2. Its first two rows tie half or all of the usable categories (those with at least 200 validation occurrences), and their labels now say so: Own channel (half tied) and All usable channels (all tied). Its column groups, After pretraining and From initialization, are your fine-tuned and from scratch.
  • The procedure and the terms each sit in one place. Section 4 opens with the whole procedure in six numbered steps (page 5), and Section 2 ends with a table of the core terms (page 4).
  • Terms are defined before use. "Treated", "usable" and "whitened dispersion", for example, are defined at or before first use, and plain language replaces terms that were not needed.
  • Each concept has one name. In the main text, the readout channel is the prediction channel, the timing has one name (from initialization), and a run under an intervention is always a treated run.
AuthorsOpenReview ↗

Response to Reviewer 4YZT (2 of 3)

W2 (groupings) and Q1. "the main text does not say which grouping the tying experiments use, and Section 5 pools results across groupings ... please state this per condition ... Reporting results per grouping would show whether the findings hold for each."

Every intervention on a real model uses one grouping, kk-means with K=50K{=}50. No run ties part-of-speech (POS) or WordNet categories, so no token is tied under two groupings. The first of Section 4's six steps now says so (page 5).

In Section 5, the association holds in every grouping alone, and the law holds in each grouping it is scored on. The submission gave the association as a count, positive in all 16 model-and-grouping combinations, and the law's held-out score per grouping partly in an appendix. The revision brings both into the main text, one row per grouping (Table 3), and labels the pooled rank partial correlation descriptive, because groupings share tokens.

No single grouping carries the pooled score. The law predicts a category's identity share, the fraction of its variance that separates its tokens, from the information its prediction uses, its type count and its number of occurrences. Fitted once on the K=20K{=}20, K=50K{=}50 and POS groupings and scored on each, its median held-out R2R^2 is 0.46, 0.38 and 0.56, on either side of the pooled 0.44 (Table 2). POS, the highest, is the only scored grouping not built from the model's representations, so the law does not depend on categories defined by clustering them.

Table 3 of the revision, one row per grouping:

GroupingCategories per modelSpearman ρ, dispersion against information (four models)Held-out R2R^2 of the Section 5 law (median, six families; exploratory)
kk-means, K=10K{=}10100.39–0.75excluded by the original noise rule (Appendix D.3)
kk-means, K=20K{=}2016–200.58–0.820.46
kk-means, K=50K{=}5034–440.62–0.850.38
Part of speech110.71–0.820.56
WordNet supersense250.63–0.83not scored on held-out families

Q3 (category-level prediction). "Does tying change category-level prediction p(S|h), which it restricts to a single direction u?"

Tying barely changes how well the model predicts which category comes next, so a loss of that prediction does not drive the comparison with the twin. It does route the category's own logit through one shared direction, your u (Proposition 1(ii)). After pretraining, the category-level loss moves by 0.031 nats with every usable category tied, about 4% of the 0.76–0.87 nats that removing those categories' choices among their tokens is predicted to cost (Appendix D.10). With half of them tied, it moves by 0.028 nats (Appendix D.5).

Q3 (projection). "Does the surviving dispersion remain if u is projected out of h? If it doesn't, the distinction may sit in the category-level prediction rather than surviving independently."

The surviving dispersion remains when u is projected out of h, so the category's own logit carries almost none of it (projection, new, exploratory; Section 4.4, Appendix D.21). The median whitened ratio to the twin moves by at most 0.006 in every model and condition. In each cell of your 2×2, we removed the states' component along the shared direction (precisely, along its part inside the states' span, the only part through which they reach the category's logit), and projected the twin's states the same way. One layer below the output the ratios are likewise unchanged.

The result is consistent with the paper's claim that the encoding persists without its own prediction. Category-level prediction through the other categories' logits remains among the possible sources Section 4.4 lists, and it too is consistent with that claim.

Table 12 of the revision, at the output layer's input (it pools the models, so its second decimal can differ from Table 1):

Cell of your 2×2Median whitened ratio to the twinAfter projection
Half the categories tied, after pretraining0.850.85
All tied, after pretraining0.850.85
Half tied, from initialization0.750.75
All tied, from initialization0.740.74
AuthorsOpenReview ↗

Response to Reviewer 4YZT (3 of 3)

Q4 (finer level). "Your experiments test broad token categories ... rather than specific facts or concepts. Do you expect the result to hold at that finer level?"

Specific month facts remain partly readable from the hidden states after the month choice is removed (new, exploratory; Section 4.3, Appendix D.21). The removal itself stays category-level: tying acts on the whole K=50K{=}50 category that holds the twelve month names in every model. In pretrained models we score 60 prompts whose next token is a month ("The Titanic sank on the 15th of"), restricted to that category's tokens.

In the two models whose twins answer most of the facts, the output gives one constant month, whereas the hidden states still answer more than half of the facts the twin gets right. We read the hidden states through the twin's month weights, with no fitting. In the other two models the twin itself answers few of the facts (37% and 18%; Table 13).

In Qwen2.5-1.5B the facts are partly recoverable even without the twin's output weights. A probe fitted on generic text that contains none of the prompts gets 34% from the tied model, about half of the 69% it gets on the twin.

Month facts in those two models (half the usable categories tied per run, two seeds pooled):

ModelTwin, all 60 factsTied model's own output (one constant answer; chance 6–8%)Its hidden states, twin's month weights, no fitting: all 60 factsOf the twin's correct facts, answered from those hidden states
Qwen2.5-1.5B84%10%62%68%
Pythia-1.4B55%10%32%55%

Q4 (prior findings). "Prior work finds that facts removed by model editing (Patil et al., 2024) or unlearning (Hong et al., 2025) remain in hidden states ... If not, does complete removal still explain the prior findings?"

For the prior findings, a residue need not mean that the removal was incomplete (Section 6). After a partial removal, a residue in the hidden states could be leftover prediction or an encoding that other predictions sustain. Here the removal is complete by construction, and the facts still remain in part. Output checks alone therefore cannot show that content is gone (Section 7).

We hope the analyses you asked for and the revised presentation lead you to consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision (revised PDF uploaded)

We thank the reviewers for their feedback. They recognize a clean intervention that rules out incomplete suppression (ELw4, 4YZT), which 4YZT calls what makes the main finding credible and the paper's main advance over prior work. They note extensive empirical validation with complementary measurements (ELw4), decodable surviving information with practical weight for unlearning (4YZT), and an interesting research question with implications for machine unlearning (XPoV).

The importance of our work lies in how unlearning is verified. Removing content can make a model stop producing it while the content's encoding remains partly readable. Our removal, complete by construction, tests this risk causally:

  • Even a complete removal leaves most of the encoding. Across the interventions, the median category keeps 56–86% of its twin's whitened dispersion (the spread between token means relative to that within each token), where the twin is an identical run without the removal. Models trained without the prediction stay 58–73% as decodable by a linear probe as their twins (see 4YZT's thread).
  • Residues found by earlier audits need not mean incomplete removal. Content removed by unlearning or editing can remain in hidden states, but after a partial removal, that residue could be leftover prediction. Complete removal rules that source out (see 4YZT's thread).
  • Unlearning should be verified in the representation as well as at the output. In a new test on 60 month facts, the model's own output is a constant guess. Its hidden states still answer more than half of the facts the twin gets right, in the two models whose twins answer most of them (half the categories treated; exploratory; Section 4.3, Table 13). We read them through the twin's month weights, with no fitting.

The revised PDF is uploaded, and every experiment and number of the submission stands. It gives the whole design in six numbered steps (page 5) and the core terms in one table (page 4), and it adds the analyses the reviewers asked for: specific facts (Section 4.3), the projection test (Section 4.4) and results for each grouping (Table 3). Each reviewer's thread gives the details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.