- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether suppressing token distinctions at the output-logit level also removes those distinctions from internal representations. Through experiments conducted both from initialization and after pretraining, with matched unconstrained controls, the authors examine whether these distinctions form or persist, what might sustain them, and whether the remaining information can be decoded. They find that substantial internal distinctions remain despite the output constraints.
Critical strengths
-
Clean intervention. Sharing output weights within each category enforces context-independent within-category probability ratios, ruling out incomplete suppression of the targeted prediction channel.
-
Extensive empirical validation. The paper examines multiple intervention settings and uses complementary measurements, including whitened dispersion and linear probes, to assess whether internal distinctions remain and are decodable.
Critical weaknesses
-
The presentation makes the paper difficult to follow and its contribution difficult to assess. 1.a. Too much terminology is introduced without a clear explanatory sequence. Definitions and operational details are scattered across sections, requiring readers to reconstruct the setup. For example, the second paragraph of the introduction introduces identity dispersion, readout channels, formation, maintenance, and unconstrained twins in rapid succession, before clearly explaining the concrete intervention. 1.b. The experimental narrative and presentation of results need clearer organization. The experimental descriptions are often difficult to parse, and experiments are presented in succession without sufficiently explaining their connections. Figure 2 and Table 1 have dense captions, while essential explanations are deferred to later sections or the appendix. Readers must repeatedly move between sections to understand the interventions, measurements, and conclusions. Simpler descriptions, concrete examples, and essential methodological details in the main text would make the evidence easier to assess.
-
The conceptual contribution and its connection to unlearning are insufficiently established. 2.a. The intervention and measurement concern different distinctions. The intervention removes output-token distinctions but leaves systematic semantic differences between their preceding contexts intact. Identity dispersion can therefore capture ordinary contextual differences, such as sunlight versus moisture, even when the corresponding output distinctions are suppressed. The paper does not sufficiently explain what the persistence of this measure reveals beyond continued contextual representation. 2.b. The motivating connection to unlearning remains underdeveloped. The experiments neither evaluate an unlearning method nor test the recovery of targeted knowledge. The paper needs to explain how retained contextual distinctions inform the motivating problem of whether forgotten content remains recoverable. Without this connection, the broader significance of the findings remains unclear.
Questions for authors
See the Weaknesses above.
6 comments
Response to Reviewer ELw4 (2 of 2)
2.a "Identity dispersion can therefore capture ordinary contextual differences ... The paper does not sufficiently explain what the persistence of this measure reveals beyond continued contextual representation."
Persistence shows more than a network representing its input: it marks a learned distinction that other predictions use and from which the removed choice among a category's tokens can be partly read back (Section 4.4, "Why persistence is informative"). Two results from the submission show this, and the revision adds one baseline:
- In a controlled task, context alone does not sustain the distinction: it survives only when another prediction reads the same attribute (in the submission). In a synthetic corpus, the treated category's next token depends on an attribute of the previous token, and two versions differ only in whether a partner category reads the same attribute or an independent one. With its own channel removed, the treated category keeps 0.47–0.53 of its twin's identity dispersion in the first version and 0.001 in the second (Figure 2b shows the same contrast on a log scale).
- In the language models, other predictions use the distinction, and the removed choice can be decoded from it (in the submission; Sections 4.3 and 4.4). In models trained without the prediction, a linear probe for the removed choice recovers 58–73% of what it recovers in the twin, counted above an untrained network. Erasing the directions that separate a treated category's tokens raises other predictions' loss 3.0 times as much as erasing random directions of the same size in the median cell, and 1.3 times when the change to the states is matched.
- The distinction is learned (baseline added in the revision). In an untrained network fed the same contexts, the fraction of a category's variance that separates its tokens is 0.02–0.05, against 0.09–0.12 after training.
For interpretability, a distinction in a model's representations therefore need not serve its own prediction, because another prediction can sustain it (Section 7).
2.b "The experiments neither evaluate an unlearning method nor test the recovery of targeted knowledge. The paper needs to explain how retained contextual distinctions inform the motivating problem of whether forgotten content remains recoverable."
The recovery test you ask for, now in the revision, answers the motivating problem: targeted knowledge stays partly recoverable after its prediction is removed (new, exploratory; Section 4.3, Appendix D.21). In every model one of the 50 categories holds the twelve month names. We remove its prediction channel from pretrained models and score 60 month facts ("The Titanic sank on the 15th of"), restricted to that category's tokens. No prompt contains its answer, so the answer the hidden states give cannot be read off the input text.
In the two models whose twins answer most of the facts, the output gives one constant month, whereas the hidden states still answer more than half of the facts the twin gets right. We read the hidden states through the twin's month weights, with no fitting. In the other two models the twin itself answers few of the facts (37% and 18%; Table 13).
In Qwen2.5-1.5B the facts are partly recoverable even without the twin's weights. A probe fitted on generic text that contains none of the prompts gets 34% from the treated model, about half of the 69% it gets on the twin.
Unlearning should therefore be verified in the representation as well as at the output, since suppressing an output does not show that the content is gone. We test the clean case of output suppression by design: the choice is gone by construction, so the surviving trace cannot be a partly removed prediction, an explanation a partial removal leaves open (Section 6). Section 7 draws this consequence, and the abstract now ends on it.
Month facts in those two models:
| Model (half the categories treated, two seeds) | Twin, all 60 facts | Treated model's own output (one constant answer; chance 6–8%) | Its hidden states, twin's month weights, no fitting: all 60 facts | Of the twin's correct facts, answered from those hidden states |
|---|---|---|---|---|
| Qwen2.5-1.5B | 84% | 10% | 62% | 68% |
| Pythia-1.4B | 55% | 10% | 32% | 55% |
Follow-up
Dear Reviewer ELw4,
Thank you for prompting a direct test of the recovery of targeted knowledge. We are glad to report the result: facts the model no longer outputs remain partly readable from its hidden states (Section 4.3, Table 13).
The discussion period is shorter than last year's and ends soon, on November 18. We would be glad if this and our earlier replies have resolved your concerns. Should anything remain open, we would welcome the chance to address it in the time left.
Reviewing is unpaid work on top of your own research, and we thank you for the thought you have given our paper.
Best regards, The authors
Response to Reviewer ELw4 (1 of 2)
Thank you for the careful reading. We are glad you note the clean intervention and the extensive empirical validation.
The introduction (page 1) now explains the intervention before it names any of the five terms you list. It walks through one example (rainy against sunny) and only then names the prediction channel (formerly the readout channel), maintenance, formation and the twin, an identical run without the removal. Identity dispersion follows in the next paragraph. Section 2 ends with a table of the core terms (page 4). We audited the main text for technical terms used before their definition, and replaced terms with plain language where they were not needed. For example, the submission used "treated", "usable" and "whitened dispersion" before defining them. Each is now defined at or before first use. We also unified terms that refer to the same concept: in the main text, the timing has one name (from initialization), and a run under an intervention is always a treated run.
We made the three changes you suggest, and Section 4 now connects the experiments as four questions, each raised by the answer before it (Sections 4.1 to 4.4). Does the encoding persist after pretraining? Does it also form without features learned before the removal? Is what survives usable? What sustains it?