- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of auditing fact deletion in language models with separable memory. The authors propose an audit that checks why a fact is still answered or not after deletion. The audit runs each fact under three conditions: with full memory, with the deletion set removed, and with all separable memory turned off. By comparing these conditions, it separates facts kept by memory, facts known by the backbone, and facts that only look deleted because the remaining memory leads to a wrong answer. Experiments on Co-LMLM and NULLs show that most facts answered after deletion are already known by the backbone, and that in Co-LMLM, some of facts that look deleted are still answered by the backbone.
Critical strengths
-
The paper studies an important problem of how to check whether a fact is deleted, which is ignored by current evaluation.
-
The method is simple and clear to understand, and can be used to audit separable memory.
-
The results are convincing and cover two systems and multiple datasets with different deletion rules.
Critical weaknesses
-
My main concern is about the novelty. The audit is adopted from prior work as the authors mention. Also, turning off retrieval to see what the backbone knows is a common practice in the RAG literature.The new parts are mainly the metrics and deletion rules and system-specific tests. Thus, I feel the contribution is somewhat incremental. Feel free to correct me if I am wrong.
-
As authors admitted, NULLs is designed to delete content specific to one source, but the audit uses general facts that are shared by many sources and are expected to be kept in the backbone. The results also suggest these facts are not really stored in the source memory. Thus, the audit on NULLs does not really test how well its deletion works, even though the authors mention this in the limitations.
-
In the paper, each system is tested with only one small model, and there is no comparison with standard unlearning methods on models without separable memory. I suggest applying the same audit to larger models and to these methods, which would show whether separable memory actually deletes facts better.
Questions for authors
See weaknesses.
We thank the reviewer for their thoughtful comments.
Novelty, given prior work and RAG practice. As the reviewer notes, turning off retrieval is common in RAG evaluations. In Co-LMLM, this corresponds to our DEL-OFF condition alone and cannot reveal when memory remaining after deletion suppresses an answer the backbone still recites. Comparing DEL-ON and DEL-OFF per fact allows us to quantify this interference. Please also see our response to Reviewer MZmS (point 1) on the contributions beyond Raeesi & Roed (2026).
The NULLs audit uses shared facts. Please see our response to Reviewer MZmS (point 2).
One small model per system and no unlearning baselines. Please see our response to Reviewer J6Fs (point 4).