- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RAQUEL, a framework that constructs diverse questions through relational databases to evaluate machine unlearning in large language models. The method first converts knowledge from the original forget and retain sets into a relational database with provenance records, then constructs a second database by deleting facts supported exclusively by the forget set. SQL queries are generated and executed on both databases. Questions are labeled “affected” or “unaffected” depending on whether the results change, and are subsequently converted into natural-language question–answer pairs.
Using TOFU, WMDP, and MUSE, the authors construct 9,450 answer-balanced evaluation examples covering factual, join, filtering, Boolean, and aggregation questions. Experiments across two 8B models, five unlearning methods, and three training seeds show that high accuracy on native retain questions can conceal capability degradation on new question forms, and that different evaluation questions can change the relative ranking of methods.
The authors also train models on a subset of RAQUEL questions and observe improved performance on held-out questions. However, subsequent unlearning can still impair retention on new questions. The main contributions are an evaluation construction pipeline grounded in provenance tracking and query execution, and an empirical analysis of limitations in existing unlearning evaluations.
Critical strengths
- Provenance tracking and execution on two databases provide an inspectable basis for evaluation construction.
RAQUEL records whether facts originate from the forget or retain set and protects facts also supported by retained sources. Comparing query results before and after deletion allows the authors to trace why a question is labeled affected.
This design is more transparent than relying solely on an LLM to determine whether a question belongs to the forget or retain set, and facilitates queries combining multiple facts. The appendix’s examination of redundantly stored facts and their effects on labels further demonstrates attention to construction consistency.
- The experiments provide meaningful new evidence of damage to retained capabilities.
The paper compares performance before and after unlearning and also uses a model trained only on the retain set as a reference. For example, in the WMDP–Llama setting, the retain-reference model achieves 26.6% accuracy on unaffected questions, whereas IDK+GD achieves 13.1%, despite retaining 97.5% accuracy on native retain questions.
The refusal analysis also shows that some methods rarely refuse native retain questions but frequently refuse new unaffected questions. This supports the need to broaden retention evaluation and provides practical value in distinguishing the side effects of unlearning strategies.
Critical weaknesses
- Unresolved scope and closed-world assumptions create a gap between database answers and the semantics of natural-language questions.
SQL execution verifies results within a given database, but does not automatically establish that the database fully represents the real-world knowledge asked about. RAQUEL’s generation prompts explicitly require questions not to mention databases, records, or sources, so some natural-language questions have a broader scope than the database actually covers.
For example, Table 11 assigns an answer of 3 to the question asking how many distinct people are associated with Ofcom. This number reflects the extracted database contents, but the question itself is not restricted to those news articles or source records. Similarly, the absence of a relationship in the database does not establish that the relationship does not exist in the natural-language sense. The vague phrase “associated with” may also encompass relationships not encoded in the database.
Moreover, deleting a piece of knowledge does not make the underlying fact false, nor does it necessarily imply that an appropriate answer should change from Yes to No. Thus, affected/unaffected labels clearly characterize database changes, but their validity as unlearning evaluation labels requires further justification. The existing small-scale manual inspection is insufficient to resolve these systematic issues.
- The scoring of affected questions and limited pre-unlearning performance constrain interpretations of unlearning quality and method rankings.
The paper treats a lower match rate against original answers on affected questions as better, but this counts successful forgetting, excessive refusal, reasoning failures, and general capability degradation alike as “improvements.” This also does not fully align with the objective stated in Section 3.1: making the unlearned model approximate the retain-only retrained model.
For example, in the TOFU–Llama setting in Table 4, the retain-reference model has a 27.5% match rate on affected questions, while several unlearning methods achieve only 5.1%–10.3%. These methods fall substantially below the reference model, which alone does not establish better unlearning.
Furthermore, in some main settings, the original and retain-reference models perform almost identically on affected questions. For WMDP–Llama, their scores are 27.8% and 27.7%, respectively, indicating that this aggregate metric barely distinguishes whether the model was trained on the forget set. The authors acknowledge this issue and attempt to mitigate it through additional training, which is constructive. Nevertheless, a more direct evaluation is needed to determine whether previously answerable target knowledge has been removed, while separating this effect from capability degradation.
Questions for authors
The following questions concern the validity of the current conclusions, and the answers may change my assessment.
- Question scope and answer validity: How do you ensure that answers to aggregation, negation, and relationship questions do not depend on an unstated database closed-world assumption? Could you separately audit whether the database answer is correct and whether the original sources sufficiently support the natural-language answer, and report results with explicitly restricted source scope?
- Unlearning metrics: Why should affected accuracy be minimized without qualification, rather than evaluated by proximity to the retain-only retrained model? Could you report results on the subset answered correctly before unlearning, distinguishing refusals, incorrect answers, partially retained answers, and behavior consistent with the retraining reference?
- Held-out set independence: When splitting the data into two halves by record in Section 4, are instances sharing the same underlying SQL query, template, or paraphrase family kept on the same side? If not, how do you distinguish generalization to new wording from generalization to new fact combinations or query structures?
- Ranking and scoring robustness: Do method rankings still change when the degree of forgetting is matched? Could you report forgetting–retention trade-off curves and specify the significance-testing procedure? For the many Boolean and counting questions, do deterministic scoring and a manually calibrated LLM judge yield consistent conclusions?
3 comments
Response to Reviewer 8MWA
Answer to Question 1
The two checks the reviewer asks for are separate in RAQUEL. Whether the database answer is correct is settled by execution, which is deterministic (Section 2.4). Whether the sources support the natural-language answer was checked by hand for 100 sampled records (Section 3.2). For each record, we read every source QA pair behind the answer and required that a value equals the source value, that a count equals the number of qualifying source statements, and that a Yes or No matches what the sources establish. 99 of the 100 records passed (all 50 affected and 49 of 50 unaffected).
We also report results with restricted scope (Table 21, Appendix D.6). On the 77.8% of records whose answers are fully traced to source facts, M_orig (Llama-3.1-8B, Table 4 setting) answers 24.0%, 21.1%, and 7.3% of unaffected questions on TOFU, WMDP, and MUSE. The finding that fine-tuned models rarely answer new questions therefore holds without absence-based answers. We thank the reviewer for this suggestion and will enlarge the human evaluation.
Answer to Question 2
Affected alignment should be evaluated by proximity to M_ret, as explained in our answer to W2 and now stated in Section 3.1. The paper and the new analyses already separate several of the cases the reviewer lists. Refusals are reported in Table 8 and Section 3.3, where IDK+GD refuses about half of the affected questions on TOFU. Behavior consistent with the retraining reference is measured by the distance to M_ret and by the post-deletion answer (Table 18). We thank the reviewer for the suggestions and will add this breakdown, with refusals, incorrect answers, and answers matching the post-deletion reference.
Answer to Question 3
The two halves share no forgotten fact and no question, and unaffected questions share no entity, but query templates are deliberately shared (Appendix A.2). Affected questions are assigned by the forgotten facts their answers depend on, and unaffected questions that name the same entity stay on the same side. A leakage check confirms that the halves share no forgotten fact, no entity among unaffected questions, and no question text. Templates are balanced across the halves, because grouping by template would place whole question types on one side.
The held-out half therefore measures generalization to new facts and entities under familiar question forms, not to new query structures, as Appendix A.2 states. Two controls separate knowledge from familiarity with the form. M_ret is trained on the same unaffected train half, so it knows the question forms equally well, and M_orig still exceeds it on held-out affected questions by 8.3 to 15.0 points (Section 4). Table 9 further reports the results by question family. We thank the reviewer for this question.
Answer to Question 4
Yes, the rankings on RAQUEL still differ when the methods are compared under the same forgetting criterion, and the revised paper now reports a forgetting and retention trade-off figure (Appendix D.7, Figure 3). For every method we select the checkpoint with the highest retain ROUGE-L among those whose forget ROUGE-L is at or below that of M_ret (Section 3.1), so all methods in Table 4 meet the same criterion. Figure 3 places each method in the plane of affected and unaffected RAQUEL alignment, at its selected checkpoint and after five epochs, with M_orig and M_ret as references. On TOFU, the methods lower affected alignment without lowering unaffected alignment. On WMDP and MUSE, both fall together, and unlearning beyond the selected checkpoint lowers unaffected alignment further for GA+GD and NPO+GD (from 23.0% to 19.4% and from 21.5% to 15.5% on WMDP). We thank the reviewer for suggesting this analysis.
For significance, we use a two-sided two-proportion z-test at the 5% level against M_ret over the questions of each column, which Section 3.1 now states. For Boolean and count questions, deterministic scoring agrees with the judge (Appendix D.6). On the answers of M_orig and M_ret (Llama-3.1-8B) that state an explicit Yes, No, or number, exact matching agrees with the judge in 96.9% of 28,134 cases (97.7% for Boolean and 96.2% for count questions). The judge is still needed for the remaining answers, which are phrased in free text. It follows a fixed rubric and never sees the model identity (Appendix E). Appendix A.1 additionally reports the answer distribution of the aggregation questions.
Gentle Reminder Before Tomorrow's Deadline (Nov 18)
Dear Reviewer 8MWA,
With the discussion period closing tomorrow (November 18), we hope you might have a brief moment to look over our response above. Please let us know if any further clarification would be helpful before the deadline.
Thank you for your time and help in improving our work!
Response to Reviewer 8MWA
We thank the reviewer for the careful reading and for questions that helped us state the scope of RAQUEL's labels more precisely. The revised paper marks all changes in blue, and the section and table numbers below refer to it.
Answer to W1
We agree that RAQUEL's answers are defined within the source corpus, and we thank the reviewer for pointing out that the paper did not state this scope explicitly. It is now stated in Sections 2.4 and 6. This scope matches our setting, since every model is fine-tuned on the source QA pairs (Section 3.1) and RAQUEL asks about the facts in that corpus. For TOFU the scope is exact, because its authors are fictitious.
The record provenance shows how much depends on absence (Appendix D.6, Table 21). In 77.8% of the released records, every row of the answer is traced to source facts. The remaining 22.2% are absence answers (No, zero, or an empty result), which hold only within the corpus, as do count totals such as the Ofcom example. Our conclusions do not rest on them. Lookup questions, whose answers are values stated in the sources, show the same findings. M_orig answers only 11.8 to 21.6% of unaffected lookup questions, and the losses of retained knowledge persist on lookup questions alone in 12 of 20 settings (Section 3.3, Table 6). In addition, every model is compared with M_ret on the same questions, so a scope mismatch affects both equally. On the phrase "associated with", Appendix B notes that most junction tables record only that two entities are associated, which limits what the questions can ask but not whether their labels are correct.
We also agree that deleting a fact does not make it false. For this reason the main score on affected questions is alignment with the original answer, read against M_ret (Section 3.1), and not agreement with the post-deletion answer. The affected label states only that the answer depends on a forget-set fact.
Answer to W2
We agree that affected alignment should be judged by its proximity to the retain model M_ret, not minimized without qualification, and we thank the reviewer for this point. The revised Section 3.1 now states this reading explicitly, and Tables 4 and 5 no longer bold the lowest value in the forget-side columns. Table 4 reports M_ret as the reference, and Section 3.3 notes that 29 of 30 endpoints end at or below M_ret and that M_orig exceeds M_ret by only 2.3 to 12.5 points on affected lookup questions, which bounds what this measure can reveal. Under the proximity criterion, the TOFU results the reviewer cites (5.1 to 16.8% against 27.5% for M_ret) indicate over-forgetting or capability loss, not better unlearning.
Two analyses give the more direct evaluation the reviewer asks for. First, after training with RAQUEL questions, M_orig exceeds M_ret on held-out affected questions by 8.3 to 15.0 points (Section 4), so the measure separates the two models. There, most methods fall far below M_ret, for example 0.4 to 23.9% against 46.6% on WMDP, and only GA+GD on TOFU approaches it (50.4% against 53.0%). Second, RAQUEL provides a post-deletion answer for every affected question by executing its query on the nullified database, which we use as an additional diagnostic (Table 18, Appendix D.3). Among the answers that no longer match the original, only SAUL gives the post-deletion answer often (29.3%), while the other methods stay below 10%, so most of their changed answers are errors or refusals.