- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper examines intermediate-layer target-answer signals after unlearning using a raw logit lens on two models and three unlearning objectives. It reports residual subtoken decodability despite output suppression and proposes a conditional analysis of layer-wise update intensity.
Critical strengths
The paper addresses the useful distinction between output suppression and internal accessibility, while appropriately limiting its claims about answer recovery.
Critical weaknesses
-
The theoretical premise is infeasible in the stated model. LayerNorm’s shift invariance implies , hence and . This contradicts the strict contraction required by Theorem 3.2, leaving the proposed explanation unsupported within its own model.
-
The empirical contribution is insufficiently established. Intermediate-layer target-token retention has already been reported by Wang et al. [1], which is not discussed. The best-subtoken statistic and unmatched irrelevant-token baseline do not adequately distinguish the reported phenomenon from readout and token-selection effects.
-
The presentation makes verification unnecessarily difficult. Several plots have small labels, and figure terminology is inconsistent with the text, including “Unaligned” for pre-unlearning checkpoints. The figures and tables should be legible and consistently labeled without requiring explanatory corrections in the surrounding prose.
-
Limited model coverage. The experiments are restricted to Phi-3 and Qwen3 at approximately 4B parameters. They do not establish whether the observed layer-wise pattern persists in newer architectures, such as the hybrid Gated DeltaNet–attention architecture of Qwen3.5. Testing a comparably sized model such as Qwen3.5-4B-Base would help distinguish a broader phenomenon from behavior specific to the evaluated checkpoints.
References
[1] Wang et al. Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models. ACL 2026. Paper
Questions for authors
-
How is Theorem 3.2’s strict contraction assumption compatible with LayerNorm’s shift invariance?
-
What substantive finding goes beyond Wang et al. [1], and does it survive controls matched for token frequency and candidate-set size?
-
Does the observed pattern persist in a contemporary architecture, such as Qwen3.5-4B-Base, under the same controlled diagnostic?
I would consider raising the rating to 2 (Weak Rejection) if the authors resolve the incompatibility in the contraction assumption, clarify the contribution beyond prior work, and show that the empirical pattern survives matched-token controls. Validation on a contemporary architecture would further strengthen the evidence.