- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks a question the unlearning literature has largely dodged: when an unlearned model matches a retrained model on standard output metrics, has it actually recovered the features retraining selects, and does that mismatch matter once training resumes? The authors make four moves: Why features matter; Deletion can require learning new features; A directional feature-recovery metric and feature mismatch to behavior under adaptation
Critical strengths
Genuinely fresh framing. Snapshot output agreement isn’t enough is a simple, compelling thesis, and the paper nails it down properly rather than leaving it as rhetoric.
Deleting an arbitrarily small fraction can require learning a feature the full model never had is an interesting idea, cleanly proved, and knocks the subtractive view of unlearning on the head. The GA runtime ratio is a tidy quantitative statement.
Critical weaknesses
The metric hangs entirely on one feature operationalisation. Everything leans on Jiang et al.’s PCA-response clustering, and it’s sensitive to K , γ_corr , κ , and the complete-linkage choice. The paper ablates k and group size and argues qualitative robustness, which helps but doesn’t close the issue. would sparse-autoencoder features, linear probes, or RSA yield the same method ordering? The paper doesn’t test an alternative feature definition, only an alternative similarity measure (CKA). That’s the single biggest vulnerability from my piont.
Theory and empirics are decoupled. The XOR construction shows deletion can require new features; it isn’t claimed to be the mechanism behind the ViT/CIFAR-100 results, and the gap between a quadratic network under gradient flow and a fine-tuned ViT is large. The theory section is valuable in its own right but reads a bit like a separate paper bolted onto the empirical study.
Questions for authors
Does the method ranking under D_feat line up with a different feature definition, say sparse autoencoders or linear-probe directions, at least on the main ViT-Tiny/CIFAR-100 setup? Even one alternative would go a fair way towards shoring up the metric’s credibility.
For the predictive scatters in Fig. 2, are the relationships actually significant with only about n≈10 method-level points, and does the ranking hold up under leave-one-method-out?
Does the partial-relearning conclusion still stand under a second protocol, like full fine-tuning rather than a fresh head on a frozen representation? The frozen-representation design is neat, but it’s also the most favourable case for the paper’s mechanism.
In the XOR model, how touchy are Thm 4.2/4.3 to the exactly-balanced scaling and the quadratic architecture? A quick note on robustness would help bridge theory and practice.